[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:47:51 dist-test-slave-65tz krb5kdc[9086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:47:51 dist-test-slave-65tz krb5kdc[9086](info): set up 2 sockets
Jun 20 14:47:51 dist-test-slave-65tz krb5kdc[9086](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 14:47:52.534287 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:47:52 dist-test-slave-65tz krb5kdc[9086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:47:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:47:52.665699 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:40283
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:40283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:52.800580 9099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:52.800906 9099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:52.800953 9099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:52.805254 9099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:52.805335 9099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:52.805356 9099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:52.805373 9099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:52.805390 9099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:47:52.811311 9099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:40283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:40283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:47:52.812789 9099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:52.814095 9099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:52.824822 9105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:52.824854 9104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:52.824836 9107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:52.825184 9099 server_base.cc:1034] running on GCE node
I20260620 14:47:52.826236 9099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:52.827713 9099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:52.828922 9099 hybrid_clock.cc:648] HybridClock initialized: now 1781966872828875 us; error 60 us; skew 500 ppm
Jun 20 14:47:52 dist-test-slave-65tz krb5kdc[9086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966872, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:52.836146 9099 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:47:52.837203 9099 webserver.cc:460] Webserver started at http://127.8.221.254:41279/ using document root <none> and password file <none>
I20260620 14:47:52.837942 9099 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:52.838038 9099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:52.838294 9099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:52.840632 9099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "8d666b99d78e4ce48f2a6a43492df8a6"
format_stamp: "Formatted at 2026-06-20 14:47:52 on dist-test-slave-65tz"
server_key: "35bce08fcde62459e59d14b952c93d9a"
server_key_iv: "581f085dc62b86fcf2dea048ed57714f"
server_key_version: "encryptionkey@0"
I20260620 14:47:52.841248 9099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "8d666b99d78e4ce48f2a6a43492df8a6"
format_stamp: "Formatted at 2026-06-20 14:47:52 on dist-test-slave-65tz"
server_key: "35bce08fcde62459e59d14b952c93d9a"
server_key_iv: "581f085dc62b86fcf2dea048ed57714f"
server_key_version: "encryptionkey@0"
I20260620 14:47:52.845702 9099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 14:47:52.848677 9114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:52.850342 9099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:47:52.850512 9099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "8d666b99d78e4ce48f2a6a43492df8a6"
format_stamp: "Formatted at 2026-06-20 14:47:52 on dist-test-slave-65tz"
server_key: "35bce08fcde62459e59d14b952c93d9a"
server_key_iv: "581f085dc62b86fcf2dea048ed57714f"
server_key_version: "encryptionkey@0"
I20260620 14:47:52.850634 9099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:52.874451 9099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:52.878021 9099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:52.878248 9099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:52.886435 9099 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:40283
I20260620 14:47:52.886495 9166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:40283 every 8 connection(s)
I20260620 14:47:52.887542 9099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:52.891400 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:52.893997 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 9099
I20260620 14:47:52.894213 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:47:52.896302 9079 external_mini_cluster.cc:1428] Setting key 1f96caa5e7cc0e73cfb73e9378e317b0
I20260620 14:47:52.900489 9167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6: Bootstrap starting.
I20260620 14:47:52.903875 9167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:52.904973 9167 log.cc:824] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:52.907526 9167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6: No bootstrap required, opened a new log
I20260620 14:47:52.911898 9167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d666b99d78e4ce48f2a6a43492df8a6" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 40283 } }
I20260620 14:47:52.912202 9167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:52.912287 9167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d666b99d78e4ce48f2a6a43492df8a6, State: Initialized, Role: FOLLOWER
I20260620 14:47:52.912940 9167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d666b99d78e4ce48f2a6a43492df8a6" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 40283 } }
I20260620 14:47:52.913142 9167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:52.913226 9167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:52.913364 9167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:52.914986 9167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d666b99d78e4ce48f2a6a43492df8a6" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 40283 } }
I20260620 14:47:52.915488 9167 leader_election.cc:302] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d666b99d78e4ce48f2a6a43492df8a6; no voters:
I20260620 14:47:52.915961 9167 leader_election.cc:288] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:52.916123 9172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:52.916455 9172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6 [term 1 LEADER]: Becoming Leader. State: Replica: 8d666b99d78e4ce48f2a6a43492df8a6, State: Running, Role: LEADER
I20260620 14:47:52.916882 9172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d666b99d78e4ce48f2a6a43492df8a6" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 40283 } }
Jun 20 14:47:52 dist-test-slave-65tz krb5kdc[9086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:47:52.917927 9167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:52.918903 9173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d666b99d78e4ce48f2a6a43492df8a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d666b99d78e4ce48f2a6a43492df8a6" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 40283 } } }
I20260620 14:47:52.919056 9173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:52.919442 9174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d666b99d78e4ce48f2a6a43492df8a6. Latest consensus state: current_term: 1 leader_uuid: "8d666b99d78e4ce48f2a6a43492df8a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d666b99d78e4ce48f2a6a43492df8a6" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 40283 } } }
I20260620 14:47:52.919559 9174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:52.920413 9180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:52.924132 9180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:52.926015 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:52.901189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.8.221.254:40283)
0620 14:47:52.901883 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:52.901909 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:52.902994 (+ 1085us) server_negotiation.cc:407] Connection header received
0620 14:47:52.906566 (+ 3572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:52.906585 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:52.906965 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:52.907327 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:52.909505 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:52.910564 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:52.911699 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:52.912157 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:52.912327 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:52.918773 (+ 6446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:52.918781 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:52.918801 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:52.918843 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:52.921376 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:52.922268 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:52.922275 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:52.922281 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:52.922422 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:52.922865 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:52.922870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:52.922872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:52.923428 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:52.923663 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:52.924497 (+ 834us) server_negotiation.cc:299] Negotiation successful
0620 14:47:52.924852 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":188,"threads_started":1}
I20260620 14:47:52.931612 9180 catalog_manager.cc:1347] Generated new cluster ID: 9caaae06fd5149c08901c4fa18c56067
I20260620 14:47:52.931701 9180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:52.942006 9180 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:52.943372 9180 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:52.950743 9180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d666b99d78e4ce48f2a6a43492df8a6: Generated new TSK 0
I20260620 14:47:52.951766 9180 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
I20260620 14:47:52.989136 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:40283
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:53.108575 9193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:53.108901 9193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:53.108951 9193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:53.112944 9193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:53.113018 9193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:53.113081 9193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:47:53.118918 9193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:40283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:47:53.120158 9193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:53.121281 9193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:53.131358 9201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:53.131417 9198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:53.131670 9199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:53.131876 9193 server_base.cc:1034] running on GCE node
I20260620 14:47:53.132385 9193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:53.133194 9193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:53.134413 9193 hybrid_clock.cc:648] HybridClock initialized: now 1781966873134394 us; error 33 us; skew 500 ppm
Jun 20 14:47:53 dist-test-slave-65tz krb5kdc[9086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966873, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:53.138000 9193 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:47:53.139099 9193 webserver.cc:460] Webserver started at http://127.8.221.193:45165/ using document root <none> and password file <none>
I20260620 14:47:53.139801 9193 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:53.139870 9193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:53.140091 9193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:53.142714 9193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "bd5847a43cf04adb95d58186d72170a6"
format_stamp: "Formatted at 2026-06-20 14:47:53 on dist-test-slave-65tz"
server_key: "57dd9e393e2f8b7f32699b06bda4d6bc"
server_key_iv: "c2e2cd3a1a43a069f2d8649497e32693"
server_key_version: "encryptionkey@0"
I20260620 14:47:53.143311 9193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "bd5847a43cf04adb95d58186d72170a6"
format_stamp: "Formatted at 2026-06-20 14:47:53 on dist-test-slave-65tz"
server_key: "57dd9e393e2f8b7f32699b06bda4d6bc"
server_key_iv: "c2e2cd3a1a43a069f2d8649497e32693"
server_key_version: "encryptionkey@0"
I20260620 14:47:53.148568 9193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:47:53.152052 9208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:53.153784 9193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:47:53.154038 9193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "bd5847a43cf04adb95d58186d72170a6"
format_stamp: "Formatted at 2026-06-20 14:47:53 on dist-test-slave-65tz"
server_key: "57dd9e393e2f8b7f32699b06bda4d6bc"
server_key_iv: "c2e2cd3a1a43a069f2d8649497e32693"
server_key_version: "encryptionkey@0"
I20260620 14:47:53.154201 9193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:53.179317 9193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:53.183410 9193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:53.183671 9193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:53.184454 9193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:53.185488 9193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:53.185539 9193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:53.185627 9193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:53.185662 9193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:53.198673 9193 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:38481
I20260620 14:47:53.198705 9321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:38481 every 8 connection(s)
I20260620 14:47:53.199908 9193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:47:53.206486 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 9193
I20260620 14:47:53.206612 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:47:53.206955 9079 external_mini_cluster.cc:1428] Setting key 7df7b4131405a1551843b12c978efc96
Jun 20 14:47:53 dist-test-slave-65tz krb5kdc[9086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966873, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:47:53.217136 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:53.202311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:57635 (local address 127.8.221.254:40283)
0620 14:47:53.202537 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:53.202544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:53.203627 (+ 1083us) server_negotiation.cc:407] Connection header received
0620 14:47:53.205077 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:53.205081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:53.205145 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:53.205257 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:53.206883 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.207556 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.209147 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.209577 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.209717 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:53.212706 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:53.212713 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:53.212719 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:53.212758 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:53.214794 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:53.215396 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:53.215401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:53.215403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:53.215498 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:53.215881 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:53.215885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:53.215887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:53.216102 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:53.216223 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:53.216741 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 14:47:53.216939 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
WARNING: no policy specified for kudu/127.8.221.194@KRBTEST.COM; defaulting to no policy
I20260620 14:47:53.218225 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:53.202855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:40283 (local address 127.8.221.193:57635)
0620 14:47:53.203417 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:53.203476 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:53.204807 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:53.205526 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:53.205534 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:53.205960 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:53.206687 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:53.206709 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.207741 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.207744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:53.209005 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:53.209015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.209743 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.209746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:53.209984 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:53.210643 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:53.210672 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:53.212520 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:53.214948 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:53.214956 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:53.214972 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:53.215273 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:53.215600 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:53.215605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:53.215607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:53.215756 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:53.216218 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:53.216225 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:53.216483 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 14:47:53.216682 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 14:47:53.217058 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":132,"threads_started":1}
I20260620 14:47:53.220003 9322 heartbeater.cc:347] Connected to a master server at 127.8.221.254:40283
I20260620 14:47:53.220293 9322 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.194@KRBTEST.COM" created.
I20260620 14:47:53.220935 9322 heartbeater.cc:511] Master 127.8.221.254:40283 requested a full tablet report, sending...
I20260620 14:47:53.223179 9128 ts_manager.cc:194] Registered new tserver with Master: bd5847a43cf04adb95d58186d72170a6 (127.8.221.193:38481)
I20260620 14:47:53.224563 9128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:57635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
I20260620 14:47:53.240666 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:40283
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:53.375223 9327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:53.375540 9327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:53.375591 9327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:53.379835 9327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:53.379899 9327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:53.379961 9327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:47:53.386130 9327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:40283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:47:53.387388 9327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:53.388674 9327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:53.399873 9332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:53.399876 9333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:53.399909 9335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:53.400270 9327 server_base.cc:1034] running on GCE node
I20260620 14:47:53.400848 9327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:53.401774 9327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:53.403033 9327 hybrid_clock.cc:648] HybridClock initialized: now 1781966873402995 us; error 55 us; skew 500 ppm
Jun 20 14:47:53 dist-test-slave-65tz krb5kdc[9086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966873, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:53.406275 9327 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:47:53.407343 9327 webserver.cc:460] Webserver started at http://127.8.221.194:42317/ using document root <none> and password file <none>
I20260620 14:47:53.408015 9327 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:53.408108 9327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:53.408377 9327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:53.410825 9327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "75aa30d1886f4dadaf5389562fb1f9c5"
format_stamp: "Formatted at 2026-06-20 14:47:53 on dist-test-slave-65tz"
server_key: "3a0bf8849af36109f12a00a30cbdb46b"
server_key_iv: "c692a73a113c81d9e8663dc427e96210"
server_key_version: "encryptionkey@0"
I20260620 14:47:53.411455 9327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "75aa30d1886f4dadaf5389562fb1f9c5"
format_stamp: "Formatted at 2026-06-20 14:47:53 on dist-test-slave-65tz"
server_key: "3a0bf8849af36109f12a00a30cbdb46b"
server_key_iv: "c692a73a113c81d9e8663dc427e96210"
server_key_version: "encryptionkey@0"
I20260620 14:47:53.415807 9327 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:47:53.418910 9342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:53.420704 9327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:47:53.420858 9327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "75aa30d1886f4dadaf5389562fb1f9c5"
format_stamp: "Formatted at 2026-06-20 14:47:53 on dist-test-slave-65tz"
server_key: "3a0bf8849af36109f12a00a30cbdb46b"
server_key_iv: "c692a73a113c81d9e8663dc427e96210"
server_key_version: "encryptionkey@0"
I20260620 14:47:53.421025 9327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:53.440398 9327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:53.443704 9327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:53.443910 9327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:53.444531 9327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:53.445433 9327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:53.445514 9327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:53.445591 9327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:53.445637 9327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:53.456418 9327 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:44727
I20260620 14:47:53.456516 9455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:44727 every 8 connection(s)
I20260620 14:47:53.457393 9327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:47:53.459014 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 9327
I20260620 14:47:53.459151 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:47:53.459465 9079 external_mini_cluster.cc:1428] Setting key 1021d2aeb0d94b23db002a8926979e41
Jun 20 14:47:53 dist-test-slave-65tz krb5kdc[9086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966873, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.195@KRBTEST.COM" created.
I20260620 14:47:53.475123 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:53.461240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:42355 (local address 127.8.221.254:40283)
0620 14:47:53.461371 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:53.461374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:53.461389 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:47:53.463965 (+ 2576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:53.463967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:53.464012 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:53.464077 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:53.465948 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.466528 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.467845 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.468086 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.468200 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:53.471178 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:53.471187 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:53.471189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:53.471212 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:53.472686 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:53.473265 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:53.473268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:53.473269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:53.473423 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:53.473981 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:53.473989 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:53.473991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:53.474199 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:53.474304 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:53.474832 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 14:47:53.474958 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:47:53.475997 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:53.460251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:40283 (local address 127.8.221.194:42355)
0620 14:47:53.460842 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:53.460906 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:53.463707 (+ 2801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:53.464289 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:53.464301 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:53.464921 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:53.465750 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:53.465770 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.466669 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.466673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:53.467713 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:53.467723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.468246 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.468251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:53.468437 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:53.469140 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:53.469157 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:53.470993 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:53.472848 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:53.472856 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:53.472873 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:53.473158 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:53.473548 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:53.473552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:53.473555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:53.473788 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:53.474314 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:53.474322 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:53.474590 (+ 268us) client_negotiation.cc:769] Sending connection context
0620 14:47:53.474777 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 14:47:53.475157 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":112,"threads_started":1}
I20260620 14:47:53.477255 9456 heartbeater.cc:347] Connected to a master server at 127.8.221.254:40283
I20260620 14:47:53.477572 9456 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:53.478300 9456 heartbeater.cc:511] Master 127.8.221.254:40283 requested a full tablet report, sending...
I20260620 14:47:53.479597 9128 ts_manager.cc:194] Registered new tserver with Master: 75aa30d1886f4dadaf5389562fb1f9c5 (127.8.221.194:44727)
I20260620 14:47:53.480207 9128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:42355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
I20260620 14:47:53.494586 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:40283
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:53.629868 9461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:53.630240 9461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:53.630331 9461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:53.635079 9461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:53.635186 9461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:53.635296 9461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:47:53.641781 9461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:40283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:47:53.643389 9461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:53.644799 9461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:53.657152 9469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:53.657155 9467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:53.657078 9466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:53.657670 9461 server_base.cc:1034] running on GCE node
I20260620 14:47:53.658418 9461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:53.659219 9461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:53.660401 9461 hybrid_clock.cc:648] HybridClock initialized: now 1781966873660381 us; error 36 us; skew 500 ppm
Jun 20 14:47:53 dist-test-slave-65tz krb5kdc[9086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966873, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:53.663431 9461 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:47:53.664361 9461 webserver.cc:460] Webserver started at http://127.8.221.195:42019/ using document root <none> and password file <none>
I20260620 14:47:53.664966 9461 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:53.665028 9461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:53.665205 9461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:53.667419 9461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "dc0d327f558a4ba388dd26223851053e"
format_stamp: "Formatted at 2026-06-20 14:47:53 on dist-test-slave-65tz"
server_key: "1fd7014f9e3a262d06eabbb7958f475b"
server_key_iv: "9e15b1fe38b24a5e966632b0aa9db51f"
server_key_version: "encryptionkey@0"
I20260620 14:47:53.667965 9461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "dc0d327f558a4ba388dd26223851053e"
format_stamp: "Formatted at 2026-06-20 14:47:53 on dist-test-slave-65tz"
server_key: "1fd7014f9e3a262d06eabbb7958f475b"
server_key_iv: "9e15b1fe38b24a5e966632b0aa9db51f"
server_key_version: "encryptionkey@0"
I20260620 14:47:53.672295 9461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:47:53.675513 9476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:53.677098 9461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:47:53.677212 9461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "dc0d327f558a4ba388dd26223851053e"
format_stamp: "Formatted at 2026-06-20 14:47:53 on dist-test-slave-65tz"
server_key: "1fd7014f9e3a262d06eabbb7958f475b"
server_key_iv: "9e15b1fe38b24a5e966632b0aa9db51f"
server_key_version: "encryptionkey@0"
I20260620 14:47:53.677309 9461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:53.690346 9461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:53.693625 9461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:53.693781 9461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:53.694386 9461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:53.695173 9461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:53.695222 9461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:53.695263 9461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:53.695278 9461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:53.706421 9461 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:33629
I20260620 14:47:53.706482 9589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:33629 every 8 connection(s)
I20260620 14:47:53.707407 9461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:47:53.711961 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 9461
I20260620 14:47:53.712210 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:47:53.712532 9079 external_mini_cluster.cc:1428] Setting key 35fd2b65b4100c072cc0919dbfa56d71
Jun 20 14:47:53 dist-test-slave-65tz krb5kdc[9086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966873, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:47:53.724300 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:53.709710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:50833 (local address 127.8.221.254:40283)
0620 14:47:53.709958 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:53.709962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:53.710882 (+ 920us) server_negotiation.cc:407] Connection header received
0620 14:47:53.713067 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:53.713071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:53.713134 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:53.713222 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:53.714895 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.715494 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.716786 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.717027 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.717143 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:53.720299 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:53.720305 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:53.720313 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:53.720338 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:53.721711 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:53.722268 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:53.722273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:53.722278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:53.722391 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:53.722864 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:53.722868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:53.722870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:53.723188 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:53.723311 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:53.723975 (+ 664us) server_negotiation.cc:299] Negotiation successful
0620 14:47:53.724109 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260620 14:47:53.725374 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:53.710117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:40283 (local address 127.8.221.195:50833)
0620 14:47:53.710656 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:53.710709 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:53.712784 (+ 2075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:53.713466 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:53.713475 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:53.714035 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:53.714725 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:53.714741 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.715665 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.715671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:53.716650 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:53.716660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.717204 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.717208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:53.717401 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:53.718138 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:53.718156 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:53.720114 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:53.721848 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:53.721856 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:53.721873 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:53.722155 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:53.722485 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:53.722488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:53.722490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:53.722751 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:53.723314 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:53.723324 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:53.723718 (+ 394us) client_negotiation.cc:769] Sending connection context
0620 14:47:53.723922 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 14:47:53.724315 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":108,"threads_started":1}
I20260620 14:47:53.726897 9590 heartbeater.cc:347] Connected to a master server at 127.8.221.254:40283
I20260620 14:47:53.727334 9590 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:53.727967 9590 heartbeater.cc:511] Master 127.8.221.254:40283 requested a full tablet report, sending...
I20260620 14:47:53.729166 9131 ts_manager.cc:194] Registered new tserver with Master: dc0d327f558a4ba388dd26223851053e (127.8.221.195:33629)
I20260620 14:47:53.729887 9131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:50833
I20260620 14:47:53.739917 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:47:53 dist-test-slave-65tz krb5kdc[9086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966873, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:47:53 dist-test-slave-65tz krb5kdc[9086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966873, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.221.193@KRBTEST.COM
I20260620 14:47:53.770489 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:53.759693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41158 (local address 127.8.221.193:38481)
0620 14:47:53.760163 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:53.760173 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:53.760307 (+ 134us) server_negotiation.cc:407] Connection header received
0620 14:47:53.760498 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:53.760504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:53.760706 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:53.760874 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:53.761571 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.762413 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.763239 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.763602 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.763720 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:53.766508 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:53.766516 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:53.766530 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:53.766552 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:53.768330 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:53.768940 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:53.768945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:53.768947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:53.769028 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:53.769355 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:53.769358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:53.769359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:53.769613 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:53.769887 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:53.770210 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 14:47:53.770350 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":127,"threads_started":1}
W20260620 14:47:53.771876 9236 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41158
Jun 20 14:47:53 dist-test-slave-65tz krb5kdc[9086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:47:53 dist-test-slave-65tz krb5kdc[9086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.193@KRBTEST.COM
I20260620 14:47:53.800227 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:53.791615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.8.221.193:38481)
0620 14:47:53.791800 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:53.791805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:53.791925 (+ 120us) server_negotiation.cc:407] Connection header received
0620 14:47:53.792111 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:53.792118 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:53.792160 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:53.792245 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:53.792950 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.793504 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.794407 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.794643 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.794753 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:53.797233 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:53.797236 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:53.797238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:53.797258 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:53.798664 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:53.799057 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:53.799062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:53.799064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:53.799139 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:53.799412 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:53.799416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:53.799418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:53.799584 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:53.799721 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:53.799945 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 14:47:53.800064 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:47:53.801647 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 9193
I20260620 14:47:53.808748 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 9327
I20260620 14:47:53.817474 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 9461
I20260620 14:47:53.823446 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 9099
2026-06-20T14:47:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2357 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:47:53 dist-test-slave-65tz krb5kdc[9604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:47:53 dist-test-slave-65tz krb5kdc[9604](info): set up 2 sockets
Jun 20 14:47:53 dist-test-slave-65tz krb5kdc[9604](info): commencing operation
krb5kdc: starting...
W20260620 14:47:54.875487 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:47:54 dist-test-slave-65tz krb5kdc[9604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:47:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:47:55.005290 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38017
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:38017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:55.134166 9617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:55.134505 9617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:55.134549 9617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:55.138873 9617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:55.138957 9617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:55.138978 9617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:55.138995 9617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:55.139034 9617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:47:55.144752 9617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:38017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:47:55.146306 9617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:55.147648 9617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:55.158035 9625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:55.158108 9622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:55.158044 9623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:55.158509 9617 server_base.cc:1034] running on GCE node
I20260620 14:47:55.159169 9617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:55.160578 9617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:55.161763 9617 hybrid_clock.cc:648] HybridClock initialized: now 1781966875161735 us; error 37 us; skew 500 ppm
Jun 20 14:47:55 dist-test-slave-65tz krb5kdc[9604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966875, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:55.165221 9617 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:47:55.166513 9617 webserver.cc:460] Webserver started at http://127.8.221.254:35535/ using document root <none> and password file <none>
I20260620 14:47:55.167222 9617 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:55.167291 9617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:55.167558 9617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:55.170146 9617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "56be761a106a498f89aa2f71abb81085"
format_stamp: "Formatted at 2026-06-20 14:47:55 on dist-test-slave-65tz"
server_key: "637df13b17a816a7c9f22196a61585ca"
server_key_iv: "dc98a038029d7e972c5ec5621615b9c2"
server_key_version: "encryptionkey@0"
I20260620 14:47:55.170827 9617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "56be761a106a498f89aa2f71abb81085"
format_stamp: "Formatted at 2026-06-20 14:47:55 on dist-test-slave-65tz"
server_key: "637df13b17a816a7c9f22196a61585ca"
server_key_iv: "dc98a038029d7e972c5ec5621615b9c2"
server_key_version: "encryptionkey@0"
I20260620 14:47:55.175505 9617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:47:55.178617 9632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:55.180184 9617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:47:55.180374 9617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "56be761a106a498f89aa2f71abb81085"
format_stamp: "Formatted at 2026-06-20 14:47:55 on dist-test-slave-65tz"
server_key: "637df13b17a816a7c9f22196a61585ca"
server_key_iv: "dc98a038029d7e972c5ec5621615b9c2"
server_key_version: "encryptionkey@0"
I20260620 14:47:55.180640 9617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:55.208000 9617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:55.211457 9617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:55.211679 9617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:55.220459 9617 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:38017
I20260620 14:47:55.220781 9684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:38017 every 8 connection(s)
I20260620 14:47:55.221642 9617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:55.222355 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 9617
I20260620 14:47:55.222474 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:47:55.222734 9079 external_mini_cluster.cc:1428] Setting key 4957db113d823c8de3d80bbc8c3fafe0
I20260620 14:47:55.225469 9685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:55.231978 9685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085: Bootstrap starting.
Jun 20 14:47:55 dist-test-slave-65tz krb5kdc[9604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:47:55.234966 9685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:55.235821 9685 log.cc:824] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:55.237697 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:55.224278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59966 (local address 127.8.221.254:38017)
0620 14:47:55.224828 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:55.224850 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:55.225080 (+ 230us) server_negotiation.cc:407] Connection header received
0620 14:47:55.226031 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:55.226051 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:55.226436 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:55.226825 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:55.227570 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:55.228659 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:55.229532 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:55.230100 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:55.230286 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:55.232812 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:55.232817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:55.232831 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:55.232860 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:55.234736 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:55.235231 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:55.235236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:55.235247 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:55.235354 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:55.235631 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:55.235636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:55.235637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:55.236089 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:55.236293 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:55.236572 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 14:47:55.236857 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":215,"threads_started":1}
I20260620 14:47:55.238149 9685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085: No bootstrap required, opened a new log
I20260620 14:47:55.242854 9685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56be761a106a498f89aa2f71abb81085" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38017 } }
I20260620 14:47:55.243127 9685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:55.243221 9685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56be761a106a498f89aa2f71abb81085, State: Initialized, Role: FOLLOWER
I20260620 14:47:55.243750 9685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56be761a106a498f89aa2f71abb81085" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38017 } }
I20260620 14:47:55.243913 9685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:55.244004 9685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:55.244108 9685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:55.245222 9685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56be761a106a498f89aa2f71abb81085" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38017 } }
I20260620 14:47:55.245618 9685 leader_election.cc:302] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56be761a106a498f89aa2f71abb81085; no voters:
I20260620 14:47:55.246079 9685 leader_election.cc:288] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:55.246253 9690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:55.246547 9690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085 [term 1 LEADER]: Becoming Leader. State: Replica: 56be761a106a498f89aa2f71abb81085, State: Running, Role: LEADER
I20260620 14:47:55.246961 9690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56be761a106a498f89aa2f71abb81085" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38017 } }
I20260620 14:47:55.247454 9685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:55.248950 9691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56be761a106a498f89aa2f71abb81085" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56be761a106a498f89aa2f71abb81085" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38017 } } }
I20260620 14:47:55.249128 9691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:55.248967 9692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56be761a106a498f89aa2f71abb81085. Latest consensus state: current_term: 1 leader_uuid: "56be761a106a498f89aa2f71abb81085" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56be761a106a498f89aa2f71abb81085" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38017 } } }
I20260620 14:47:55.249434 9692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085 [sys.catalog]: This master's current role is: LEADER
W20260620 14:47:55.253028 9705 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:47:55.253124 9705 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:47:55.253222 9702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:55.254040 9702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:55.261179 9702 catalog_manager.cc:1347] Generated new cluster ID: 21626f0d593248deba24e95901f0e15c
I20260620 14:47:55.261260 9702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:55.271924 9702 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:55.273465 9702 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:55.279312 9702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56be761a106a498f89aa2f71abb81085: Generated new TSK 0
I20260620 14:47:55.280189 9702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
I20260620 14:47:55.321401 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38017
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:55.473234 9711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:55.473560 9711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:55.473611 9711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:55.477736 9711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:55.477816 9711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:55.477950 9711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:47:55.483595 9711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38017
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:47:55.484989 9711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:55.486318 9711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:55.497296 9716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:55.497315 9717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:55.497359 9719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:55.497316 9711 server_base.cc:1034] running on GCE node
I20260620 14:47:55.498092 9711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:55.498888 9711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:55.500066 9711 hybrid_clock.cc:648] HybridClock initialized: now 1781966875500033 us; error 50 us; skew 500 ppm
Jun 20 14:47:55 dist-test-slave-65tz krb5kdc[9604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966875, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:55.503072 9711 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:47:55.504019 9711 webserver.cc:460] Webserver started at http://127.8.221.193:43163/ using document root <none> and password file <none>
I20260620 14:47:55.504667 9711 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:55.504765 9711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:55.505007 9711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:55.507259 9711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "6903c791aa414166a567eef6e3a77914"
format_stamp: "Formatted at 2026-06-20 14:47:55 on dist-test-slave-65tz"
server_key: "72455b055edf32b46add15b5c0ca60e2"
server_key_iv: "688315019a6422e31a74a371cb4b8370"
server_key_version: "encryptionkey@0"
I20260620 14:47:55.507848 9711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "6903c791aa414166a567eef6e3a77914"
format_stamp: "Formatted at 2026-06-20 14:47:55 on dist-test-slave-65tz"
server_key: "72455b055edf32b46add15b5c0ca60e2"
server_key_iv: "688315019a6422e31a74a371cb4b8370"
server_key_version: "encryptionkey@0"
I20260620 14:47:55.512099 9711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:47:55.514976 9726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:55.516525 9711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:47:55.516681 9711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "6903c791aa414166a567eef6e3a77914"
format_stamp: "Formatted at 2026-06-20 14:47:55 on dist-test-slave-65tz"
server_key: "72455b055edf32b46add15b5c0ca60e2"
server_key_iv: "688315019a6422e31a74a371cb4b8370"
server_key_version: "encryptionkey@0"
I20260620 14:47:55.516800 9711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:55.540987 9711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:55.544092 9711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:55.544297 9711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:55.544898 9711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:55.545739 9711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:55.545842 9711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:55.545934 9711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:55.545981 9711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:55.556641 9711 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:44279
I20260620 14:47:55.556668 9839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:44279 every 8 connection(s)
I20260620 14:47:55.557536 9711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:47:55.559512 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 9711
I20260620 14:47:55.559664 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:47:55.559998 9079 external_mini_cluster.cc:1428] Setting key 586f712f74f5189e40f73f9feae04ac8
Jun 20 14:47:55 dist-test-slave-65tz krb5kdc[9604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966875, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.194@KRBTEST.COM" created.
I20260620 14:47:55.574225 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:55.559816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:37537 (local address 127.8.221.254:38017)
0620 14:47:55.559988 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:55.559994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:55.560996 (+ 1002us) server_negotiation.cc:407] Connection header received
0620 14:47:55.563091 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:55.563093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:55.563136 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:55.563212 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:55.564760 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:55.565356 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:55.566702 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:55.566946 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:55.567074 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:55.570333 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:55.570340 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:55.570346 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:55.570381 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:55.571759 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:55.572255 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:55.572257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:55.572258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:55.572329 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:55.572689 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:55.572693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:55.572695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:55.572943 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:55.573090 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:55.573739 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 14:47:55.573947 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:47:55.574831 9842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:55.560241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38017 (local address 127.8.221.193:37537)
0620 14:47:55.560798 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:55.560853 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:55.562815 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:55.563446 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:55.563459 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:55.564025 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:55.564597 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:55.564612 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:55.565491 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:55.565495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:55.566575 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:55.566585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:55.567173 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:55.567182 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:55.567492 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:55.568241 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:55.568264 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:55.570131 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:55.571871 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:55.571876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:55.571888 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:55.572155 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:55.572411 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:55.572413 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:55.572415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:55.572571 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:55.573105 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:55.573111 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:55.573373 (+ 262us) client_negotiation.cc:769] Sending connection context
0620 14:47:55.573548 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 14:47:55.573925 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":154,"threads_started":1}
I20260620 14:47:55.576316 9840 heartbeater.cc:347] Connected to a master server at 127.8.221.254:38017
I20260620 14:47:55.576622 9840 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:55.577195 9840 heartbeater.cc:511] Master 127.8.221.254:38017 requested a full tablet report, sending...
I20260620 14:47:55.579023 9645 ts_manager.cc:194] Registered new tserver with Master: 6903c791aa414166a567eef6e3a77914 (127.8.221.193:44279)
I20260620 14:47:55.580284 9645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:37537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
I20260620 14:47:55.589263 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38017
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:55.704039 9845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:55.704296 9845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:55.704340 9845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:55.708148 9845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:55.708200 9845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:55.708256 9845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:47:55.713286 9845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38017
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:47:55.714346 9845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:55.715368 9845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:55.725387 9851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:55.725463 9845 server_base.cc:1034] running on GCE node
W20260620 14:47:55.725415 9853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:55.725363 9850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:55.726109 9845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:55.726694 9845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:55.727857 9845 hybrid_clock.cc:648] HybridClock initialized: now 1781966875727819 us; error 58 us; skew 500 ppm
Jun 20 14:47:55 dist-test-slave-65tz krb5kdc[9604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966875, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:55.730726 9845 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:47:55.731607 9845 webserver.cc:460] Webserver started at http://127.8.221.194:46749/ using document root <none> and password file <none>
I20260620 14:47:55.732142 9845 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:55.732235 9845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:55.732456 9845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:55.734447 9845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "9b3b6e2d6b1844f6bf6a85b2edd944d1"
format_stamp: "Formatted at 2026-06-20 14:47:55 on dist-test-slave-65tz"
server_key: "f58babc08ebc5bc982ca35d94ff251c3"
server_key_iv: "183489770e6bf4ffbab18b73da918ec9"
server_key_version: "encryptionkey@0"
I20260620 14:47:55.734985 9845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "9b3b6e2d6b1844f6bf6a85b2edd944d1"
format_stamp: "Formatted at 2026-06-20 14:47:55 on dist-test-slave-65tz"
server_key: "f58babc08ebc5bc982ca35d94ff251c3"
server_key_iv: "183489770e6bf4ffbab18b73da918ec9"
server_key_version: "encryptionkey@0"
I20260620 14:47:55.738965 9845 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:47:55.741493 9860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:55.742861 9845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:47:55.742997 9845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "9b3b6e2d6b1844f6bf6a85b2edd944d1"
format_stamp: "Formatted at 2026-06-20 14:47:55 on dist-test-slave-65tz"
server_key: "f58babc08ebc5bc982ca35d94ff251c3"
server_key_iv: "183489770e6bf4ffbab18b73da918ec9"
server_key_version: "encryptionkey@0"
I20260620 14:47:55.743101 9845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:55.757370 9845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:55.760422 9845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:55.760607 9845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:55.761150 9845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:55.762020 9845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:55.762090 9845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:55.762162 9845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:55.762219 9845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:55.772023 9845 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:35263
I20260620 14:47:55.772133 9973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:35263 every 8 connection(s)
I20260620 14:47:55.772930 9845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:47:55.776294 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 9845
I20260620 14:47:55.776415 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:47:55.776695 9079 external_mini_cluster.cc:1428] Setting key dfa181eaa49671e3a8e01ff365d87be9
Jun 20 14:47:55 dist-test-slave-65tz krb5kdc[9604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966875, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.195@KRBTEST.COM; defaulting to no policy
I20260620 14:47:55.790016 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:55.775154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:44213 (local address 127.8.221.254:38017)
0620 14:47:55.775374 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:55.775378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:55.776178 (+ 800us) server_negotiation.cc:407] Connection header received
0620 14:47:55.778130 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:55.778137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:55.778199 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:55.778321 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:55.779905 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:55.780513 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:55.782390 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:55.782599 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:55.782679 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:55.786081 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:55.786086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:55.786091 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:55.786124 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:55.787819 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:55.788416 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:55.788419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:55.788421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:55.788495 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:55.788841 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:55.788844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:55.788845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:55.789081 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:55.789164 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:55.789675 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 14:47:55.789786 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.195@KRBTEST.COM" created.
I20260620 14:47:55.790987 9976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:55.775459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38017 (local address 127.8.221.194:44213)
0620 14:47:55.775948 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:55.776031 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:55.777854 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:55.778543 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:55.778554 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:55.779035 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:55.779735 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:55.779752 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:55.780650 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:55.780658 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:55.782248 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:55.782262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:55.782769 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:55.782776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:55.783027 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:55.783735 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:55.783762 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:55.785884 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:55.787968 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:55.787976 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:55.787993 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:55.788288 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:55.788581 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:55.788583 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:55.788584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:55.788726 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:55.789179 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:55.789184 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:55.789423 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 14:47:55.789571 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 14:47:55.789975 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":110,"threads_started":1}
I20260620 14:47:55.792230 9974 heartbeater.cc:347] Connected to a master server at 127.8.221.254:38017
I20260620 14:47:55.792565 9974 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:55.793164 9974 heartbeater.cc:511] Master 127.8.221.254:38017 requested a full tablet report, sending...
I20260620 14:47:55.794421 9645 ts_manager.cc:194] Registered new tserver with Master: 9b3b6e2d6b1844f6bf6a85b2edd944d1 (127.8.221.194:35263)
I20260620 14:47:55.795013 9645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:44213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
I20260620 14:47:55.808861 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38017
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:55.934475 9979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:55.934821 9979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:55.934906 9979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:55.939095 9979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:55.939188 9979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:55.939281 9979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:47:55.944882 9979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38017
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:47:55.946324 9979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:55.948047 9979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:55.959864 9987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:55.960006 9979 server_base.cc:1034] running on GCE node
W20260620 14:47:55.959867 9984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:55.959867 9985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:55.960695 9979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:55.961475 9979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:55.962661 9979 hybrid_clock.cc:648] HybridClock initialized: now 1781966875962624 us; error 56 us; skew 500 ppm
Jun 20 14:47:55 dist-test-slave-65tz krb5kdc[9604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966875, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:55.965678 9979 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:47:55.966620 9979 webserver.cc:460] Webserver started at http://127.8.221.195:46763/ using document root <none> and password file <none>
I20260620 14:47:55.967204 9979 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:55.967264 9979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:55.967432 9979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:55.969542 9979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "069d2f47aa8c4cb9b9c4c3dd8f336b4b"
format_stamp: "Formatted at 2026-06-20 14:47:55 on dist-test-slave-65tz"
server_key: "f7c6e35ada6499227ea2afcad47cba12"
server_key_iv: "dbeb52f90f318ab8a0f03970c28568e3"
server_key_version: "encryptionkey@0"
I20260620 14:47:55.970197 9979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "069d2f47aa8c4cb9b9c4c3dd8f336b4b"
format_stamp: "Formatted at 2026-06-20 14:47:55 on dist-test-slave-65tz"
server_key: "f7c6e35ada6499227ea2afcad47cba12"
server_key_iv: "dbeb52f90f318ab8a0f03970c28568e3"
server_key_version: "encryptionkey@0"
I20260620 14:47:55.974435 9979 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:47:55.977121 9994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:55.978654 9979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:47:55.978793 9979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "069d2f47aa8c4cb9b9c4c3dd8f336b4b"
format_stamp: "Formatted at 2026-06-20 14:47:55 on dist-test-slave-65tz"
server_key: "f7c6e35ada6499227ea2afcad47cba12"
server_key_iv: "dbeb52f90f318ab8a0f03970c28568e3"
server_key_version: "encryptionkey@0"
I20260620 14:47:55.978904 9979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:56.008803 9979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:56.011976 9979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:56.012176 9979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:56.012784 9979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:56.013620 9979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:56.013693 9979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:56.013756 9979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:56.013834 9979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:56.024019 9979 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:39469
I20260620 14:47:56.024055 10107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:39469 every 8 connection(s)
I20260620 14:47:56.024883 9979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:47:56.026008 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 9979
I20260620 14:47:56.026150 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:47:56.026420 9079 external_mini_cluster.cc:1428] Setting key ddecc970f04eb308548885e0fe569038
Jun 20 14:47:56 dist-test-slave-65tz krb5kdc[9604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966875, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:47:56.039817 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:56.027307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:53083 (local address 127.8.221.254:38017)
0620 14:47:56.027472 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:56.027477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:56.029026 (+ 1549us) server_negotiation.cc:407] Connection header received
0620 14:47:56.030403 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:56.030406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:56.030456 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:56.030530 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:56.031690 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.032215 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.033337 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.033562 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.033671 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:56.036120 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:56.036124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:56.036128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:56.036153 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:56.037456 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:56.037998 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:56.038002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:56.038004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:56.038077 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:56.038433 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:56.038436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:56.038438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:56.038602 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:56.038716 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:56.039445 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 14:47:56.039632 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:47:56.040660 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:56.027820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38017 (local address 127.8.221.195:53083)
0620 14:47:56.028864 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:56.028924 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:56.030196 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:56.030710 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:56.030718 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:56.031050 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:56.031536 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:56.031549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.032337 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.032340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:56.033207 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:56.033216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.033713 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.033717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:56.033911 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:56.034552 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:56.034567 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:56.035961 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:56.037582 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:56.037587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:56.037601 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:56.037899 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:56.038193 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:56.038196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:56.038198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:56.038332 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:56.038704 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:56.038710 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:56.038938 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 14:47:56.039141 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 14:47:56.039425 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":148,"threads_started":1}
I20260620 14:47:56.041853 10108 heartbeater.cc:347] Connected to a master server at 127.8.221.254:38017
I20260620 14:47:56.042191 10108 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:56.042825 10108 heartbeater.cc:511] Master 127.8.221.254:38017 requested a full tablet report, sending...
I20260620 14:47:56.044018 9645 ts_manager.cc:194] Registered new tserver with Master: 069d2f47aa8c4cb9b9c4c3dd8f336b4b (127.8.221.195:39469)
I20260620 14:47:56.044687 9645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:53083
I20260620 14:47:56.052932 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:47:56.064152 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:56.056605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59976 (local address 127.8.221.254:38017)
0620 14:47:56.056747 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:56.056751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:56.056819 (+ 68us) server_negotiation.cc:407] Connection header received
0620 14:47:56.056965 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:56.056967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:56.057015 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:56.057162 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:56.057902 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.058434 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.059340 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.059574 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.059667 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:56.061072 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:56.061075 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:56.061077 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:56.061096 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:56.062715 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:56.063094 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:56.063096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:56.063097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:56.063167 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:56.063408 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:56.063410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:56.063411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:56.063623 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:56.063734 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:56.063927 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 14:47:56.064021 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:47:56.068764 9645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:47:56.071931 9645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:47:56.087392 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:56.081081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:39469 (local address 127.0.0.1:56170)
0620 14:47:56.081519 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:56.081568 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:56.081689 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:56.082741 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:56.082747 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:56.082770 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:56.082936 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:56.082943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.084427 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.084429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:56.085646 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:56.085656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.086838 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.086841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:56.086948 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:56.086983 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:47:56.087098 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:47:56.087221 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":117,"threads_started":1}
I20260620 14:47:56.087540 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:56.081407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:35263 (local address 127.0.0.1:41976)
0620 14:47:56.081994 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:56.082006 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:56.082087 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:56.082933 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:56.082935 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:56.082950 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:56.083046 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:56.083051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.084634 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.084638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:56.085915 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:56.085923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.087118 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.087122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:56.087220 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:56.087240 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:47:56.087302 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:47:56.087364 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":94,"threads_started":1}
I20260620 14:47:56.087885 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:56.082054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41976 (local address 127.8.221.194:35263)
0620 14:47:56.082407 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:56.082421 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:56.082446 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:47:56.082520 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:56.082524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:56.082781 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:56.083186 (+ 405us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:56.083324 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.084477 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.086072 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.086996 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.087131 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:56.087297 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:56.087560 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 14:47:56.087690 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":185,"threads_started":1}
I20260620 14:47:56.088199 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:56.081295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56170 (local address 127.8.221.195:39469)
0620 14:47:56.081950 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:56.081968 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:56.081997 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:47:56.082094 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:56.082273 (+ 179us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:56.082547 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:56.082751 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:56.083057 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.084284 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.085869 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.086713 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.086843 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:56.086994 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:56.087604 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 14:47:56.087707 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":253,"threads_started":1}
I20260620 14:47:56.089442 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:56.082498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55900 (local address 127.8.221.193:44279)
0620 14:47:56.083270 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:56.083280 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:56.083624 (+ 344us) server_negotiation.cc:407] Connection header received
0620 14:47:56.083706 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:56.083711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:56.083917 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:56.084066 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:56.084362 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.085656 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.087157 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.088277 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.088389 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:56.088520 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:56.088887 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 14:47:56.089287 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":137,"threads_started":1}
I20260620 14:47:56.089984 10122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:56.082042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:44279 (local address 127.0.0.1:55900)
0620 14:47:56.083415 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:56.083430 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:56.083532 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:56.084056 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:56.084060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:56.084079 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:56.084175 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:56.084181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.085845 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.085849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:56.087017 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:56.087029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.088586 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.088590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:56.088692 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:56.088715 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:47:56.088798 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:47:56.088892 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1307,"thread_start_us":100,"threads_started":1}
I20260620 14:47:56.091643 10042 tablet_service.cc:1467] Processing CreateTablet for tablet ce431644f7d8493b962f8459ababd609 (DEFAULT_TABLE table=test-table [id=8b9726e8aaaf43c7b888f4f8754915ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:56.091643 9908 tablet_service.cc:1467] Processing CreateTablet for tablet ce431644f7d8493b962f8459ababd609 (DEFAULT_TABLE table=test-table [id=8b9726e8aaaf43c7b888f4f8754915ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:56.093022 9908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce431644f7d8493b962f8459ababd609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:56.093022 10042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce431644f7d8493b962f8459ababd609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:56.094863 9774 tablet_service.cc:1467] Processing CreateTablet for tablet ce431644f7d8493b962f8459ababd609 (DEFAULT_TABLE table=test-table [id=8b9726e8aaaf43c7b888f4f8754915ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:56.095930 9774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce431644f7d8493b962f8459ababd609. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:56.102404 10126 tablet_bootstrap.cc:492] T ce431644f7d8493b962f8459ababd609 P 069d2f47aa8c4cb9b9c4c3dd8f336b4b: Bootstrap starting.
I20260620 14:47:56.103358 10125 tablet_bootstrap.cc:492] T ce431644f7d8493b962f8459ababd609 P 9b3b6e2d6b1844f6bf6a85b2edd944d1: Bootstrap starting.
I20260620 14:47:56.103957 10127 tablet_bootstrap.cc:492] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914: Bootstrap starting.
I20260620 14:47:56.106130 10126 tablet_bootstrap.cc:654] T ce431644f7d8493b962f8459ababd609 P 069d2f47aa8c4cb9b9c4c3dd8f336b4b: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:56.106289 10127 tablet_bootstrap.cc:654] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:56.106487 10125 tablet_bootstrap.cc:654] T ce431644f7d8493b962f8459ababd609 P 9b3b6e2d6b1844f6bf6a85b2edd944d1: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:56.107172 10127 log.cc:824] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:56.107160 10126 log.cc:824] T ce431644f7d8493b962f8459ababd609 P 069d2f47aa8c4cb9b9c4c3dd8f336b4b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:56.107270 10125 log.cc:824] T ce431644f7d8493b962f8459ababd609 P 9b3b6e2d6b1844f6bf6a85b2edd944d1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:56.109330 10126 tablet_bootstrap.cc:492] T ce431644f7d8493b962f8459ababd609 P 069d2f47aa8c4cb9b9c4c3dd8f336b4b: No bootstrap required, opened a new log
I20260620 14:47:56.109544 10126 ts_tablet_manager.cc:1397] T ce431644f7d8493b962f8459ababd609 P 069d2f47aa8c4cb9b9c4c3dd8f336b4b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 14:47:56.109601 10127 tablet_bootstrap.cc:492] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914: No bootstrap required, opened a new log
I20260620 14:47:56.109602 10125 tablet_bootstrap.cc:492] T ce431644f7d8493b962f8459ababd609 P 9b3b6e2d6b1844f6bf6a85b2edd944d1: No bootstrap required, opened a new log
I20260620 14:47:56.109854 10127 ts_tablet_manager.cc:1397] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:47:56.109854 10125 ts_tablet_manager.cc:1397] T ce431644f7d8493b962f8459ababd609 P 9b3b6e2d6b1844f6bf6a85b2edd944d1: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260620 14:47:56.113349 10126 raft_consensus.cc:348] T ce431644f7d8493b962f8459ababd609 P 069d2f47aa8c4cb9b9c4c3dd8f336b4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d2f47aa8c4cb9b9c4c3dd8f336b4b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39469 } } peers { permanent_uuid: "6903c791aa414166a567eef6e3a77914" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 44279 } } peers { permanent_uuid: "9b3b6e2d6b1844f6bf6a85b2edd944d1" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 35263 } }
I20260620 14:47:56.113619 10126 raft_consensus.cc:374] T ce431644f7d8493b962f8459ababd609 P 069d2f47aa8c4cb9b9c4c3dd8f336b4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:56.113690 10126 raft_consensus.cc:729] T ce431644f7d8493b962f8459ababd609 P 069d2f47aa8c4cb9b9c4c3dd8f336b4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 069d2f47aa8c4cb9b9c4c3dd8f336b4b, State: Initialized, Role: FOLLOWER
I20260620 14:47:56.114197 10126 consensus_queue.cc:260] T ce431644f7d8493b962f8459ababd609 P 069d2f47aa8c4cb9b9c4c3dd8f336b4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d2f47aa8c4cb9b9c4c3dd8f336b4b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39469 } } peers { permanent_uuid: "6903c791aa414166a567eef6e3a77914" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 44279 } } peers { permanent_uuid: "9b3b6e2d6b1844f6bf6a85b2edd944d1" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 35263 } }
I20260620 14:47:56.114238 10125 raft_consensus.cc:348] T ce431644f7d8493b962f8459ababd609 P 9b3b6e2d6b1844f6bf6a85b2edd944d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d2f47aa8c4cb9b9c4c3dd8f336b4b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39469 } } peers { permanent_uuid: "6903c791aa414166a567eef6e3a77914" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 44279 } } peers { permanent_uuid: "9b3b6e2d6b1844f6bf6a85b2edd944d1" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 35263 } }
I20260620 14:47:56.114439 10127 raft_consensus.cc:348] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d2f47aa8c4cb9b9c4c3dd8f336b4b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39469 } } peers { permanent_uuid: "6903c791aa414166a567eef6e3a77914" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 44279 } } peers { permanent_uuid: "9b3b6e2d6b1844f6bf6a85b2edd944d1" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 35263 } }
I20260620 14:47:56.114648 10125 raft_consensus.cc:374] T ce431644f7d8493b962f8459ababd609 P 9b3b6e2d6b1844f6bf6a85b2edd944d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:56.114749 10125 raft_consensus.cc:729] T ce431644f7d8493b962f8459ababd609 P 9b3b6e2d6b1844f6bf6a85b2edd944d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b3b6e2d6b1844f6bf6a85b2edd944d1, State: Initialized, Role: FOLLOWER
I20260620 14:47:56.114960 10108 heartbeater.cc:503] Master 127.8.221.254:38017 was elected leader, sending a full tablet report...
I20260620 14:47:56.115263 10126 ts_tablet_manager.cc:1428] T ce431644f7d8493b962f8459ababd609 P 069d2f47aa8c4cb9b9c4c3dd8f336b4b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:47:56.114712 10127 raft_consensus.cc:374] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:56.115268 10125 consensus_queue.cc:260] T ce431644f7d8493b962f8459ababd609 P 9b3b6e2d6b1844f6bf6a85b2edd944d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d2f47aa8c4cb9b9c4c3dd8f336b4b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39469 } } peers { permanent_uuid: "6903c791aa414166a567eef6e3a77914" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 44279 } } peers { permanent_uuid: "9b3b6e2d6b1844f6bf6a85b2edd944d1" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 35263 } }
I20260620 14:47:56.115429 10127 raft_consensus.cc:729] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6903c791aa414166a567eef6e3a77914, State: Initialized, Role: FOLLOWER
I20260620 14:47:56.115897 10127 consensus_queue.cc:260] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d2f47aa8c4cb9b9c4c3dd8f336b4b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39469 } } peers { permanent_uuid: "6903c791aa414166a567eef6e3a77914" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 44279 } } peers { permanent_uuid: "9b3b6e2d6b1844f6bf6a85b2edd944d1" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 35263 } }
I20260620 14:47:56.116118 9974 heartbeater.cc:503] Master 127.8.221.254:38017 was elected leader, sending a full tablet report...
I20260620 14:47:56.116572 10125 ts_tablet_manager.cc:1428] T ce431644f7d8493b962f8459ababd609 P 9b3b6e2d6b1844f6bf6a85b2edd944d1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 14:47:56.117069 10127 ts_tablet_manager.cc:1428] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 14:47:56.117411 9840 heartbeater.cc:503] Master 127.8.221.254:38017 was elected leader, sending a full tablet report...
I20260620 14:47:56.172854 10133 raft_consensus.cc:482] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:47:56.173082 10133 raft_consensus.cc:504] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d2f47aa8c4cb9b9c4c3dd8f336b4b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39469 } } peers { permanent_uuid: "6903c791aa414166a567eef6e3a77914" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 44279 } } peers { permanent_uuid: "9b3b6e2d6b1844f6bf6a85b2edd944d1" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 35263 } }
I20260620 14:47:56.174378 10133 leader_election.cc:288] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 069d2f47aa8c4cb9b9c4c3dd8f336b4b (127.8.221.195:39469), 9b3b6e2d6b1844f6bf6a85b2edd944d1 (127.8.221.194:35263)
I20260620 14:47:56.179426 10134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:56.174616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:39469 (local address 127.8.221.193:34157)
0620 14:47:56.175051 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:56.175066 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:56.175208 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:56.175613 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:56.175616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:56.175659 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:56.175866 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:56.175873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.176972 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.176976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:56.178228 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:56.178238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.178967 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.178972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:56.179080 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:56.179136 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 14:47:56.179220 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:47:56.179299 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":86,"spinlock_wait_cycles":9856,"thread_start_us":157,"threads_started":1}
I20260620 14:47:56.179551 10121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:56.174815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:34157 (local address 127.8.221.195:39469)
0620 14:47:56.175057 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:56.175062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:56.175182 (+ 120us) server_negotiation.cc:407] Connection header received
0620 14:47:56.175391 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:56.175394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:56.175466 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:56.175563 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:56.176010 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.176852 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.178366 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.178823 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.178933 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:56.178981 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:56.179258 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 14:47:56.179364 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260620 14:47:56.179733 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:56.174920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:45135 (local address 127.8.221.194:35263)
0620 14:47:56.175051 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:56.175058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:56.175392 (+ 334us) server_negotiation.cc:407] Connection header received
0620 14:47:56.175462 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:56.175465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:56.175529 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:56.175614 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:56.176010 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.176848 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.178473 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.178950 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.179074 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:56.179138 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:56.179529 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 14:47:56.179621 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:47:56.179746 10135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:56.174830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:35263 (local address 127.8.221.193:45135)
0620 14:47:56.175260 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:56.175282 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:56.175379 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:56.175637 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:56.175640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:56.175663 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:56.175866 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:56.175873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.177009 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.177019 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:56.178285 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:56.178320 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.179093 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.179096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:56.179296 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:56.179389 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 14:47:56.179477 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:47:56.179563 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":87,"spinlock_wait_cycles":3328,"thread_start_us":151,"threads_started":1}
I20260620 14:47:56.180449 10062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce431644f7d8493b962f8459ababd609" candidate_uuid: "6903c791aa414166a567eef6e3a77914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "069d2f47aa8c4cb9b9c4c3dd8f336b4b" is_pre_election: true
I20260620 14:47:56.180512 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce431644f7d8493b962f8459ababd609" candidate_uuid: "6903c791aa414166a567eef6e3a77914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b3b6e2d6b1844f6bf6a85b2edd944d1" is_pre_election: true
I20260620 14:47:56.180823 9928 raft_consensus.cc:2454] T ce431644f7d8493b962f8459ababd609 P 9b3b6e2d6b1844f6bf6a85b2edd944d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6903c791aa414166a567eef6e3a77914 in term 0.
I20260620 14:47:56.180861 10062 raft_consensus.cc:2454] T ce431644f7d8493b962f8459ababd609 P 069d2f47aa8c4cb9b9c4c3dd8f336b4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6903c791aa414166a567eef6e3a77914 in term 0.
I20260620 14:47:56.181396 9730 leader_election.cc:302] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6903c791aa414166a567eef6e3a77914, 9b3b6e2d6b1844f6bf6a85b2edd944d1; no voters:
I20260620 14:47:56.181728 10133 raft_consensus.cc:2781] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:47:56.181877 10133 raft_consensus.cc:482] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:47:56.181939 10133 raft_consensus.cc:3037] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:56.183219 10133 raft_consensus.cc:504] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d2f47aa8c4cb9b9c4c3dd8f336b4b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39469 } } peers { permanent_uuid: "6903c791aa414166a567eef6e3a77914" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 44279 } } peers { permanent_uuid: "9b3b6e2d6b1844f6bf6a85b2edd944d1" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 35263 } }
I20260620 14:47:56.183640 10133 leader_election.cc:288] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914 [CANDIDATE]: Term 1 election: Requested vote from peers 069d2f47aa8c4cb9b9c4c3dd8f336b4b (127.8.221.195:39469), 9b3b6e2d6b1844f6bf6a85b2edd944d1 (127.8.221.194:35263)
I20260620 14:47:56.183995 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce431644f7d8493b962f8459ababd609" candidate_uuid: "6903c791aa414166a567eef6e3a77914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b3b6e2d6b1844f6bf6a85b2edd944d1"
I20260620 14:47:56.184028 10062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce431644f7d8493b962f8459ababd609" candidate_uuid: "6903c791aa414166a567eef6e3a77914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "069d2f47aa8c4cb9b9c4c3dd8f336b4b"
I20260620 14:47:56.184106 9928 raft_consensus.cc:3037] T ce431644f7d8493b962f8459ababd609 P 9b3b6e2d6b1844f6bf6a85b2edd944d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:56.184187 10062 raft_consensus.cc:3037] T ce431644f7d8493b962f8459ababd609 P 069d2f47aa8c4cb9b9c4c3dd8f336b4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:56.185328 9928 raft_consensus.cc:2454] T ce431644f7d8493b962f8459ababd609 P 9b3b6e2d6b1844f6bf6a85b2edd944d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6903c791aa414166a567eef6e3a77914 in term 1.
I20260620 14:47:56.185657 9730 leader_election.cc:302] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6903c791aa414166a567eef6e3a77914, 9b3b6e2d6b1844f6bf6a85b2edd944d1; no voters:
I20260620 14:47:56.185681 10062 raft_consensus.cc:2454] T ce431644f7d8493b962f8459ababd609 P 069d2f47aa8c4cb9b9c4c3dd8f336b4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6903c791aa414166a567eef6e3a77914 in term 1.
I20260620 14:47:56.185974 10133 raft_consensus.cc:2781] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:56.186268 10133 raft_consensus.cc:686] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914 [term 1 LEADER]: Becoming Leader. State: Replica: 6903c791aa414166a567eef6e3a77914, State: Running, Role: LEADER
I20260620 14:47:56.186636 10133 consensus_queue.cc:237] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d2f47aa8c4cb9b9c4c3dd8f336b4b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39469 } } peers { permanent_uuid: "6903c791aa414166a567eef6e3a77914" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 44279 } } peers { permanent_uuid: "9b3b6e2d6b1844f6bf6a85b2edd944d1" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 35263 } }
I20260620 14:47:56.189785 9648 catalog_manager.cc:5495] T ce431644f7d8493b962f8459ababd609 P 6903c791aa414166a567eef6e3a77914 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6903c791aa414166a567eef6e3a77914 (127.8.221.193). New cstate: current_term: 1 leader_uuid: "6903c791aa414166a567eef6e3a77914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069d2f47aa8c4cb9b9c4c3dd8f336b4b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6903c791aa414166a567eef6e3a77914" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 44279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b3b6e2d6b1844f6bf6a85b2edd944d1" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 35263 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:47:56 dist-test-slave-65tz krb5kdc[9604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.193@KRBTEST.COM
I20260620 14:47:56.214815 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:56.203975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55908 (local address 127.8.221.193:44279)
0620 14:47:56.204182 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:56.204188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:56.204388 (+ 200us) server_negotiation.cc:407] Connection header received
0620 14:47:56.204588 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:56.204592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:56.204659 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:56.204765 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:56.205595 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.206286 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.207259 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.207518 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.207637 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:56.210745 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:56.210765 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:56.210786 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:56.210810 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:56.212687 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:56.213164 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:56.213178 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:56.213179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:56.213260 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:56.213638 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:56.213641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:56.213643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:56.214012 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:56.214290 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:56.214454 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 14:47:56.214591 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
Jun 20 14:47:56 dist-test-slave-65tz krb5kdc[9604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966876, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:47:56 dist-test-slave-65tz krb5kdc[9604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966876, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.221.193@KRBTEST.COM
I20260620 14:47:56.246531 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:56.237349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55914 (local address 127.8.221.193:44279)
0620 14:47:56.237551 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:56.237559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:56.237727 (+ 168us) server_negotiation.cc:407] Connection header received
0620 14:47:56.237957 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:56.237960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:56.238016 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:56.238100 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:56.238823 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.239407 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.240323 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.240566 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.240663 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:56.243245 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:56.243248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:56.243251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:56.243282 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:56.244699 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:56.245109 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:56.245113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:56.245114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:56.245195 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:56.245446 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:56.245449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:56.245450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:56.245627 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:56.245764 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:56.246108 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 14:47:56.246292 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 14:47:56 dist-test-slave-65tz krb5kdc[9604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:47:56 dist-test-slave-65tz krb5kdc[9604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.193@KRBTEST.COM
W20260620 14:47:56.274025 9975 tablet.cc:2369] T ce431644f7d8493b962f8459ababd609 P 9b3b6e2d6b1844f6bf6a85b2edd944d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:47:56.275475 10124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:56.265634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.8.221.193:44279)
0620 14:47:56.265901 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:56.265908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:56.266086 (+ 178us) server_negotiation.cc:407] Connection header received
0620 14:47:56.266214 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:56.266217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:56.266277 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:56.266415 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:56.267102 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.267682 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.268589 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:56.268805 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:56.268892 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:56.271885 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:56.271890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:56.271892 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:56.271925 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:56.273430 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:56.274039 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:56.274043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:56.274045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:56.274169 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:56.274515 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:56.274518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:56.274520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:56.274725 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:56.274917 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:56.275137 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 14:47:56.275277 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260620 14:47:56.275904 10109 tablet.cc:2369] T ce431644f7d8493b962f8459ababd609 P 069d2f47aa8c4cb9b9c4c3dd8f336b4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:47:56.276944 9754 tablet_service.cc:2908] Rejecting scan request for tablet ce431644f7d8493b962f8459ababd609: Uninitialized: clean time has not yet been initialized
I20260620 14:47:56.279300 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 9711
I20260620 14:47:56.287110 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 9845
I20260620 14:47:56.293630 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 9979
I20260620 14:47:56.302292 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 9617
2026-06-20T14:47:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2481 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:47:56 dist-test-slave-65tz krb5kdc[10150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:47:56 dist-test-slave-65tz krb5kdc[10150](info): set up 2 sockets
Jun 20 14:47:56 dist-test-slave-65tz krb5kdc[10150](info): commencing operation
krb5kdc: starting...
W20260620 14:47:57.420224 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:47:57 dist-test-slave-65tz krb5kdc[10150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:47:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:47:57.556706 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:34971
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:43137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:34971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:57.670395 10163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:57.670694 10163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:57.670764 10163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:57.674739 10163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:57.674831 10163 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:47:57.674862 10163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:57.674906 10163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:57.674942 10163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:57.674978 10163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:47:57.680331 10163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:43137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:34971
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:34971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:47:57.681423 10163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:57.682475 10163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:57.691548 10169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:57.691542 10168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:57.691715 10163 server_base.cc:1034] running on GCE node
W20260620 14:47:57.691519 10171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:57.692502 10163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:57.693645 10163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:57.694849 10163 hybrid_clock.cc:648] HybridClock initialized: now 1781966877694833 us; error 39 us; skew 500 ppm
Jun 20 14:47:57 dist-test-slave-65tz krb5kdc[10150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966877, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:57.697657 10163 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:47:57.698603 10163 webserver.cc:460] Webserver started at http://127.8.221.254:41583/ using document root <none> and password file <none>
I20260620 14:47:57.699124 10163 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:57.699198 10163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:57.699360 10163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:57.701301 10163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "d70fbd4037324718a68aaf4dcccb973c"
format_stamp: "Formatted at 2026-06-20 14:47:57 on dist-test-slave-65tz"
server_key: "9b327367a9c95994d6682ddb0b6c6a9b"
server_key_iv: "8f1c2c5943f91d6211423d7b05373110"
server_key_version: "encryptionkey@0"
I20260620 14:47:57.701786 10163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "d70fbd4037324718a68aaf4dcccb973c"
format_stamp: "Formatted at 2026-06-20 14:47:57 on dist-test-slave-65tz"
server_key: "9b327367a9c95994d6682ddb0b6c6a9b"
server_key_iv: "8f1c2c5943f91d6211423d7b05373110"
server_key_version: "encryptionkey@0"
I20260620 14:47:57.705559 10163 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:47:57.708117 10178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:57.709535 10163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:47:57.709656 10163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "d70fbd4037324718a68aaf4dcccb973c"
format_stamp: "Formatted at 2026-06-20 14:47:57 on dist-test-slave-65tz"
server_key: "9b327367a9c95994d6682ddb0b6c6a9b"
server_key_iv: "8f1c2c5943f91d6211423d7b05373110"
server_key_version: "encryptionkey@0"
I20260620 14:47:57.709746 10163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:57.731644 10163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:57.735093 10163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:57.735281 10163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:57.744592 10163 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:34971
I20260620 14:47:57.744704 10240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:34971 every 8 connection(s)
I20260620 14:47:57.745684 10163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:57.749258 10241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:57.754269 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 10163
I20260620 14:47:57.754376 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:47:57.754670 9079 external_mini_cluster.cc:1428] Setting key b118594d83e373befc4207f1214640b1
I20260620 14:47:57.756340 10241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c: Bootstrap starting.
I20260620 14:47:57.759402 10241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:57.760397 10241 log.cc:824] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:57.763015 10241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c: No bootstrap required, opened a new log
Jun 20 14:47:57 dist-test-slave-65tz krb5kdc[10150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:47:57.768002 10241 raft_consensus.cc:348] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70fbd4037324718a68aaf4dcccb973c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 34971 } }
I20260620 14:47:57.768311 10241 raft_consensus.cc:374] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:57.768361 10241 raft_consensus.cc:729] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d70fbd4037324718a68aaf4dcccb973c, State: Initialized, Role: FOLLOWER
I20260620 14:47:57.768833 10241 consensus_queue.cc:260] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70fbd4037324718a68aaf4dcccb973c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 34971 } }
I20260620 14:47:57.769023 10241 raft_consensus.cc:388] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:57.769098 10241 raft_consensus.cc:482] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:57.769205 10241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:57.770539 10241 raft_consensus.cc:504] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70fbd4037324718a68aaf4dcccb973c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 34971 } }
I20260620 14:47:57.770915 10241 leader_election.cc:302] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d70fbd4037324718a68aaf4dcccb973c; no voters:
I20260620 14:47:57.771301 10241 leader_election.cc:288] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:57.771550 10246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:57.772007 10246 raft_consensus.cc:686] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c [term 1 LEADER]: Becoming Leader. State: Replica: d70fbd4037324718a68aaf4dcccb973c, State: Running, Role: LEADER
I20260620 14:47:57.772357 10246 consensus_queue.cc:237] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70fbd4037324718a68aaf4dcccb973c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 34971 } }
I20260620 14:47:57.772652 10241 sys_catalog.cc:564] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:57.772006 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:57.756405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40000 (local address 127.8.221.254:34971)
0620 14:47:57.756903 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:57.756919 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:57.756955 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:47:57.757766 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:57.757784 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:57.758183 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:57.758490 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:57.759639 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:57.760873 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:57.762022 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:57.762695 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:57.762867 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:57.766097 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:57.766104 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:57.766121 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:57.766163 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:57.768419 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:57.768958 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:57.768965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:57.768972 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:57.769134 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:57.769585 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:57.769589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:57.769590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:57.770138 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:57.770348 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:57.770681 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 14:47:57.770987 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":151,"threads_started":1}
I20260620 14:47:57.774161 10247 sys_catalog.cc:455] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d70fbd4037324718a68aaf4dcccb973c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70fbd4037324718a68aaf4dcccb973c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 34971 } } }
I20260620 14:47:57.774330 10247 sys_catalog.cc:458] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:57.774876 10248 sys_catalog.cc:455] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d70fbd4037324718a68aaf4dcccb973c. Latest consensus state: current_term: 1 leader_uuid: "d70fbd4037324718a68aaf4dcccb973c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70fbd4037324718a68aaf4dcccb973c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 34971 } } }
I20260620 14:47:57.776479 10248 sys_catalog.cc:458] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c [sys.catalog]: This master's current role is: LEADER
W20260620 14:47:57.778267 10261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:47:57.778363 10261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:47:57.778482 10259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:57.779625 10259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:57.786501 10259 catalog_manager.cc:1347] Generated new cluster ID: c58150816e484aa5bf2b01b4196f2d04
I20260620 14:47:57.786609 10259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:57.800470 10259 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:57.802001 10259 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:57.811417 10259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c: Generated new TSK 0
I20260620 14:47:57.812388 10259 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
I20260620 14:47:57.856863 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:34971
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:43137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:57.977216 10267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:57.977489 10267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:57.977542 10267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:57.981662 10267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:57.981734 10267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:57.981772 10267 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:47:57.981880 10267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:47:57.987282 10267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:43137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.221.254:34971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:47:57.988423 10267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:57.989450 10267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:58.000324 10273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:58.000586 10267 server_base.cc:1034] running on GCE node
W20260620 14:47:58.000324 10275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:58.000342 10272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:58.001149 10267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:58.001935 10267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:58.003130 10267 hybrid_clock.cc:648] HybridClock initialized: now 1781966878003128 us; error 69 us; skew 500 ppm
Jun 20 14:47:58 dist-test-slave-65tz krb5kdc[10150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966878, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:58.015033 10267 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:47:58.015990 10267 webserver.cc:460] Webserver started at http://127.8.221.193:46727/ using document root <none> and password file <none>
I20260620 14:47:58.016523 10267 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:58.016594 10267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:58.016758 10267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:58.018898 10267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "ab77ffc47683479bb5334a330bb75596"
format_stamp: "Formatted at 2026-06-20 14:47:58 on dist-test-slave-65tz"
server_key: "ca2f68342866aaec04a3a628139961d6"
server_key_iv: "b87dbb7b44c32d0e4177ae5b8f4b4782"
server_key_version: "encryptionkey@0"
I20260620 14:47:58.019415 10267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "ab77ffc47683479bb5334a330bb75596"
format_stamp: "Formatted at 2026-06-20 14:47:58 on dist-test-slave-65tz"
server_key: "ca2f68342866aaec04a3a628139961d6"
server_key_iv: "b87dbb7b44c32d0e4177ae5b8f4b4782"
server_key_version: "encryptionkey@0"
I20260620 14:47:58.023296 10267 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:47:58.025869 10282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:58.027199 10267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:47:58.027299 10267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "ab77ffc47683479bb5334a330bb75596"
format_stamp: "Formatted at 2026-06-20 14:47:58 on dist-test-slave-65tz"
server_key: "ca2f68342866aaec04a3a628139961d6"
server_key_iv: "b87dbb7b44c32d0e4177ae5b8f4b4782"
server_key_version: "encryptionkey@0"
I20260620 14:47:58.027437 10267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:58.044795 10267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:58.047864 10267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:58.048061 10267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:58.049701 10267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:58.049765 10267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:58.049853 10267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:58.049894 10267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:47:58 dist-test-slave-65tz krb5kdc[10150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966878, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:47:58.061473 10267 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:33287
I20260620 14:47:58.061578 10397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:33287 every 8 connection(s)
I20260620 14:47:58.062548 10267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:47:58.063719 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 10267
I20260620 14:47:58.063827 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:47:58.064116 9079 external_mini_cluster.cc:1428] Setting key e005421e024c80c62e898c0239b34bfc
I20260620 14:47:58.066422 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.050539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:48375 (local address 127.8.221.254:34971)
0620 14:47:58.050706 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:58.050711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:58.052506 (+ 1795us) server_negotiation.cc:407] Connection header received
0620 14:47:58.054030 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:58.054033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:58.054099 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:58.054192 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:58.055654 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.056244 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.057568 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.057933 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.058049 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.061697 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:58.061703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:58.061707 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:58.061743 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:58.063792 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:58.064510 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:58.064517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:58.064520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:58.064650 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:58.065055 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:58.065060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:58.065062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:58.065285 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:58.065397 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:58.066088 (+ 691us) server_negotiation.cc:299] Negotiation successful
0620 14:47:58.066238 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:47:58.067519 10291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.051534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:34971 (local address 127.8.221.193:48375)
0620 14:47:58.052283 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:58.052368 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:58.053726 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:58.054465 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:58.054473 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:58.054904 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:58.055488 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.055503 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.056376 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.056380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.057441 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.057451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.058101 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.058105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.058298 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.059079 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:58.059106 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:58.061477 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:58.063939 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:58.063946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:58.063964 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:58.064355 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:58.064773 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:58.064778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:58.064780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:58.064946 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:58.065402 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:58.065411 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:58.065754 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 14:47:58.066033 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 14:47:58.066413 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":143,"threads_started":1}
I20260620 14:47:58.068920 10398 heartbeater.cc:347] Connected to a master server at 127.8.221.254:34971
I20260620 14:47:58.069110 10398 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:58.069701 10398 heartbeater.cc:511] Master 127.8.221.254:34971 requested a full tablet report, sending...
I20260620 14:47:58.071331 10192 ts_manager.cc:194] Registered new tserver with Master: ab77ffc47683479bb5334a330bb75596 (127.8.221.193:33287)
I20260620 14:47:58.072897 10192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:48375
WARNING: no policy specified for kudu/127.8.221.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.194@KRBTEST.COM" created.
I20260620 14:47:58.081617 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.071833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:34971 (local address 127.8.221.193:52931)
0620 14:47:58.072191 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:58.072207 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:58.072314 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:58.072612 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:58.072615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:58.072839 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:58.072980 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.072990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.075191 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.075193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.075840 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.075849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.076372 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.076376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.076498 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.076985 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:58.076998 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:58.077370 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:58.079951 (+ 2581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:58.079955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:58.079957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:58.080210 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:58.080556 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:58.080560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:58.080562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:58.080645 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:58.081031 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:58.081034 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:58.081137 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 14:47:58.081248 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:47:58.081406 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":117,"threads_started":1}
I20260620 14:47:58.081609 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.071945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:52931 (local address 127.8.221.254:34971)
0620 14:47:58.072090 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:58.072097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:58.072300 (+ 203us) server_negotiation.cc:407] Connection header received
0620 14:47:58.072407 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:58.072413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:58.072474 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:58.072580 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:58.073228 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.074185 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.075958 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.076181 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.076363 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.077547 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:58.077551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:58.077554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:58.077580 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:58.079794 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:58.080337 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:58.080342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:58.080344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:58.080442 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:58.080738 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:58.080743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:58.080745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:58.080933 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:58.081058 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:58.081299 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 14:47:58.081436 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
I20260620 14:47:58.096537 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:34971
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:43137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:58.228794 10407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:58.229152 10407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:58.229244 10407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:58.233762 10407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:58.233889 10407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:58.233968 10407 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:47:58.234051 10407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:47:58.239766 10407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:43137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.221.254:34971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:47:58.241117 10407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:58.242408 10407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:58.254016 10415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:58.253978 10412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:58.253994 10413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:58.254385 10407 server_base.cc:1034] running on GCE node
I20260620 14:47:58.254944 10407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:58.255712 10407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:58.256891 10407 hybrid_clock.cc:648] HybridClock initialized: now 1781966878256867 us; error 40 us; skew 500 ppm
Jun 20 14:47:58 dist-test-slave-65tz krb5kdc[10150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966878, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:58.260028 10407 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:47:58.261202 10407 webserver.cc:460] Webserver started at http://127.8.221.194:36797/ using document root <none> and password file <none>
I20260620 14:47:58.261929 10407 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:58.262023 10407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:58.262296 10407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:58.264730 10407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806"
format_stamp: "Formatted at 2026-06-20 14:47:58 on dist-test-slave-65tz"
server_key: "ef927abbbd33f26838ecab91bfc959e8"
server_key_iv: "c6e311aa965a891b7f42052bf727e76a"
server_key_version: "encryptionkey@0"
I20260620 14:47:58.265372 10407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806"
format_stamp: "Formatted at 2026-06-20 14:47:58 on dist-test-slave-65tz"
server_key: "ef927abbbd33f26838ecab91bfc959e8"
server_key_iv: "c6e311aa965a891b7f42052bf727e76a"
server_key_version: "encryptionkey@0"
I20260620 14:47:58.270200 10407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 14:47:58.273506 10422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:58.275853 10407 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:47:58.275995 10407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806"
format_stamp: "Formatted at 2026-06-20 14:47:58 on dist-test-slave-65tz"
server_key: "ef927abbbd33f26838ecab91bfc959e8"
server_key_iv: "c6e311aa965a891b7f42052bf727e76a"
server_key_version: "encryptionkey@0"
I20260620 14:47:58.276150 10407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:58.288319 10407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:58.291857 10407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:58.292002 10407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:58.293457 10407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:58.293509 10407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:58.293550 10407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:58.293603 10407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:58.305107 10407 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:43339
Jun 20 14:47:58 dist-test-slave-65tz krb5kdc[10150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966878, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:47:58.306269 10407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:47:58.306654 10537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:43339 every 8 connection(s)
I20260620 14:47:58.311576 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.296564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:37881 (local address 127.8.221.254:34971)
0620 14:47:58.296752 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:58.296759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:58.296782 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:47:58.297719 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:58.297726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:58.297819 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:58.297888 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:58.299778 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.300977 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.302451 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.302862 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.302984 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.306437 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:58.306441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:58.306444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:58.306476 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:58.309114 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:58.309684 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:58.309687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:58.309689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:58.309770 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:58.310287 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:58.310294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:58.310296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:58.310513 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:58.310646 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:58.311233 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 14:47:58.311372 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 14:47:58.312597 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.295286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:34971 (local address 127.8.221.194:37881)
0620 14:47:58.295955 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:58.296017 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:58.297413 (+ 1396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:58.298345 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:58.298357 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:58.298868 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:58.299601 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.299623 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.301130 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.301137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.302302 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.302315 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.303022 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.303026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.303289 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.304093 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:58.304116 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:58.306208 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:58.309257 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:58.309263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:58.309274 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:58.309573 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:58.309916 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:58.309921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:58.309923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:58.310091 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:58.310646 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:58.310655 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:58.310941 (+ 286us) client_negotiation.cc:769] Sending connection context
0620 14:47:58.311187 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 14:47:58.311585 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":215,"threads_started":1}
I20260620 14:47:58.313846 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 10407
I20260620 14:47:58.313977 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:47:58.314198 10538 heartbeater.cc:347] Connected to a master server at 127.8.221.254:34971
I20260620 14:47:58.314344 9079 external_mini_cluster.cc:1428] Setting key c5b850919719d84212c681bb95e373c2
I20260620 14:47:58.314414 10538 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:58.315035 10538 heartbeater.cc:511] Master 127.8.221.254:34971 requested a full tablet report, sending...
I20260620 14:47:58.316243 10192 ts_manager.cc:194] Registered new tserver with Master: 051f6d84bf0e4bd0b1cf4aa46f71f806 (127.8.221.194:43339)
I20260620 14:47:58.317075 10192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:37881
WARNING: no policy specified for kudu/127.8.221.195@KRBTEST.COM; defaulting to no policy
I20260620 14:47:58.325330 10544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.316841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:34971 (local address 127.8.221.194:41305)
0620 14:47:58.317132 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:58.317145 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:58.317267 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:58.317782 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:58.317785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:58.318054 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:58.318173 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.318180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.319297 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.319300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.319915 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.319923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.320590 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.320593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.320669 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.321250 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:58.321270 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:58.321614 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:58.323655 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:58.323658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:58.323659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:58.323870 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:58.324174 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:58.324176 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:58.324177 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:58.324257 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:58.324821 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:58.324823 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:58.324896 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 14:47:58.324976 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:47:58.325092 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":120,"threads_started":1}
I20260620 14:47:58.325420 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.317385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:41305 (local address 127.8.221.254:34971)
0620 14:47:58.317524 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:58.317531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:58.317547 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:47:58.317588 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:58.317594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:58.317647 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:58.317747 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:58.318310 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.319148 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.320042 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.320445 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.320607 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.321769 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:58.321774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:58.321778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:58.321837 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:58.323524 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:58.323975 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:58.323979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:58.323980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:58.324067 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:58.324360 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:58.324363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:58.324366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:58.324693 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:58.324807 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:58.325075 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:47:58.325211 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
I20260620 14:47:58.344306 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:34971
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:43137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:58.460469 10547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:58.460716 10547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:58.460758 10547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:58.464834 10547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:58.464890 10547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:58.464927 10547 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:47:58.464957 10547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:47:58.470148 10547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:43137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.221.254:34971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:47:58.471064 10547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:58.471992 10547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:58.482910 10552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:58.483062 10555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:58.483273 10553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:58.483748 10547 server_base.cc:1034] running on GCE node
I20260620 14:47:58.484282 10547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:58.485086 10547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:58.486322 10547 hybrid_clock.cc:648] HybridClock initialized: now 1781966878486290 us; error 51 us; skew 500 ppm
Jun 20 14:47:58 dist-test-slave-65tz krb5kdc[10150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966878, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:58.489558 10547 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:47:58.490667 10547 webserver.cc:460] Webserver started at http://127.8.221.195:39341/ using document root <none> and password file <none>
I20260620 14:47:58.491324 10547 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:58.491418 10547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:58.491683 10547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:58.494122 10547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "68216c4e67024739a5a2539c05322c7e"
format_stamp: "Formatted at 2026-06-20 14:47:58 on dist-test-slave-65tz"
server_key: "c77d68de7a21e50768599a8c0dd45881"
server_key_iv: "7ce6f9c5fd8d82b54d4bbb58e553bd78"
server_key_version: "encryptionkey@0"
I20260620 14:47:58.494763 10547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "68216c4e67024739a5a2539c05322c7e"
format_stamp: "Formatted at 2026-06-20 14:47:58 on dist-test-slave-65tz"
server_key: "c77d68de7a21e50768599a8c0dd45881"
server_key_iv: "7ce6f9c5fd8d82b54d4bbb58e553bd78"
server_key_version: "encryptionkey@0"
I20260620 14:47:58.499068 10547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:47:58.501868 10562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:58.503234 10547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:47:58.503383 10547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "68216c4e67024739a5a2539c05322c7e"
format_stamp: "Formatted at 2026-06-20 14:47:58 on dist-test-slave-65tz"
server_key: "c77d68de7a21e50768599a8c0dd45881"
server_key_iv: "7ce6f9c5fd8d82b54d4bbb58e553bd78"
server_key_version: "encryptionkey@0"
I20260620 14:47:58.503500 10547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:58.530226 10547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:58.533676 10547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:58.533959 10547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:58.535465 10547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:58.535542 10547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:58.535617 10547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:58.535669 10547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:47:58 dist-test-slave-65tz krb5kdc[10150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966878, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:47:58.547250 10547 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:43197
I20260620 14:47:58.547415 10677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:43197 every 8 connection(s)
I20260620 14:47:58.548456 10547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:47:58.551438 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.536802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:45517 (local address 127.8.221.254:34971)
0620 14:47:58.537009 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:58.537014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:58.538048 (+ 1034us) server_negotiation.cc:407] Connection header received
0620 14:47:58.539426 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:58.539430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:58.539495 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:58.539600 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:58.541322 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.542239 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.543777 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.544033 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.544153 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.547376 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:58.547380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:58.547384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:58.547420 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:58.549136 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:58.549763 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:58.549767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:58.549768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:58.549934 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:58.550272 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:58.550275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:58.550276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:58.550478 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:58.550567 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:58.551092 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 14:47:58.551216 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 14:47:58.551614 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 10547
I20260620 14:47:58.551759 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:47:58.552098 9079 external_mini_cluster.cc:1428] Setting key ed5742f4500bcf2d4273b0a627fe72ab
I20260620 14:47:58.552279 10571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.537241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:34971 (local address 127.8.221.195:45517)
0620 14:47:58.537851 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:58.537908 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:58.539182 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:58.539799 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:58.539806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:58.540352 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:58.541094 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.541117 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.542413 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.542417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.543641 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.543651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.544207 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.544212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.544395 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.545099 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:58.545125 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:58.547144 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:58.549320 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:58.549327 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:58.549341 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:58.549628 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:58.550042 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:58.550044 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:58.550046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:58.550175 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:58.550578 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:58.550583 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:58.550829 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 14:47:58.551009 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 14:47:58.551276 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":169,"threads_started":1}
I20260620 14:47:58.553783 10678 heartbeater.cc:347] Connected to a master server at 127.8.221.254:34971
I20260620 14:47:58.553968 10678 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:58.554598 10678 heartbeater.cc:511] Master 127.8.221.254:34971 requested a full tablet report, sending...
I20260620 14:47:58.555778 10192 ts_manager.cc:194] Registered new tserver with Master: 68216c4e67024739a5a2539c05322c7e (127.8.221.195:43197)
I20260620 14:47:58.556541 10192 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:45517
I20260620 14:47:58.563822 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.556365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:34971 (local address 127.8.221.195:56675)
0620 14:47:58.556707 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:58.556722 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:58.556866 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:58.557157 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:58.557160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:58.557430 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:58.557602 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.557610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.558450 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.558453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.559252 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.559261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.559704 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.559707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.559774 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.560381 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:58.560393 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:58.560710 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:58.562371 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:58.562374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:58.562375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:58.562568 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:58.562891 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:58.562894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:58.562896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:58.563008 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:58.563377 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:58.563379 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:58.563457 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 14:47:58.563548 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:47:58.563670 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":22,"thread_start_us":120,"threads_started":1}
I20260620 14:47:58.563829 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.556553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:56675 (local address 127.8.221.254:34971)
0620 14:47:58.556682 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:58.556687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:58.556816 (+ 129us) server_negotiation.cc:407] Connection header received
0620 14:47:58.556959 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:58.556962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:58.557013 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:58.557111 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:58.557733 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.558306 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.559369 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.559598 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.559685 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.560848 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:58.560850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:58.560852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:58.560874 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:58.562264 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:58.562667 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:58.562670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:58.562671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:58.562746 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:58.563114 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:58.563117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:58.563118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:58.563278 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:58.563363 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:58.563617 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 14:47:58.563712 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:47:58.566128 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:47:58 dist-test-slave-65tz krb5kdc[10150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966878, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:47:58 dist-test-slave-65tz krb5kdc[10150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966878, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:47:58.593833 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.585303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40006 (local address 127.8.221.254:34971)
0620 14:47:58.585449 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:58.585455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:58.585471 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:47:58.585520 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:58.585524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:58.585575 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:58.585673 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:58.586489 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.586992 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.587842 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.588053 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.588131 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.590688 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:58.590691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:58.590693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:58.590715 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:58.592133 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:58.592580 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:58.592584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:58.592585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:58.592661 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:58.592942 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:58.592946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:58.592947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:58.593125 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:58.593261 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:58.593514 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 14:47:58.593632 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:47:58.596946 10192 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:47:58.599978 10192 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:47:58.616495 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.609873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:43197 (local address 127.0.0.1:39786)
0620 14:47:58.610726 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:58.610741 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:58.610832 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:58.611419 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:58.611426 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:58.611462 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:58.611712 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.611719 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.612984 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.612988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.614580 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.614589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.615860 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.615864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.616071 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.616102 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:47:58.616169 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:47:58.616321 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":725,"spinlock_wait_cycles":6272,"thread_start_us":95,"threads_started":1}
I20260620 14:47:58.616492 10700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.610875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.8.221.193:33287)
0620 14:47:58.611696 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:58.611703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:58.611728 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:47:58.611807 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:58.611812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:58.612019 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:58.612177 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:58.612454 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.613543 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.614942 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.615681 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.615807 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.615955 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:58.616268 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 14:47:58.616366 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":91,"threads_started":1}
I20260620 14:47:58.616508 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.610086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39786 (local address 127.8.221.195:43197)
0620 14:47:58.610672 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:58.610686 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:58.610921 (+ 235us) server_negotiation.cc:407] Connection header received
0620 14:47:58.611000 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:58.611005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:58.611234 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:58.611405 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:58.611829 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.612856 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.614715 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.615714 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.616003 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.616193 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:58.616249 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 14:47:58.616345 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":121,"threads_started":1}
I20260620 14:47:58.616847 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.610719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:33287 (local address 127.0.0.1:40886)
0620 14:47:58.611170 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:58.611184 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:58.611276 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:58.612217 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:58.612221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:58.612241 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:58.612325 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.612331 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.613672 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.613675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.614805 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.614816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.615806 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.615809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.615918 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.615954 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:47:58.616159 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 14:47:58.616714 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":447,"thread_start_us":88,"threads_started":1}
I20260620 14:47:58.616992 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.609873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:43339 (local address 127.0.0.1:34772)
0620 14:47:58.610269 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:58.610326 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:58.610474 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:58.611420 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:58.611426 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:58.611463 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:58.611689 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.611698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.613362 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.613366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.614766 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.614777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.615860 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.615864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.616008 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.616035 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:47:58.616101 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:47:58.616290 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":59,"spinlock_wait_cycles":5760,"thread_start_us":78,"threads_started":1}
I20260620 14:47:58.617580 10697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.610118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34772 (local address 127.8.221.194:43339)
0620 14:47:58.610876 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:58.610889 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:58.610914 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:47:58.610999 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:58.611005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:58.611234 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:58.611405 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:58.611856 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.613074 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.614906 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.615714 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.616388 (+ 674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.616541 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:58.617295 (+ 754us) server_negotiation.cc:299] Negotiation successful
0620 14:47:58.617412 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":145,"threads_started":1}
I20260620 14:47:58.620185 10332 tablet_service.cc:1467] Processing CreateTablet for tablet 42abd86bdc974f40a798282888502398 (DEFAULT_TABLE table=test-table [id=7f0370ef622f475f8b03aba43eaa95fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:58.620797 10612 tablet_service.cc:1467] Processing CreateTablet for tablet 42abd86bdc974f40a798282888502398 (DEFAULT_TABLE table=test-table [id=7f0370ef622f475f8b03aba43eaa95fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:58.620796 10472 tablet_service.cc:1467] Processing CreateTablet for tablet 42abd86bdc974f40a798282888502398 (DEFAULT_TABLE table=test-table [id=7f0370ef622f475f8b03aba43eaa95fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:58.621506 10332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42abd86bdc974f40a798282888502398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:58.622046 10472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42abd86bdc974f40a798282888502398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:58.622046 10612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42abd86bdc974f40a798282888502398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:58.628427 10701 tablet_bootstrap.cc:492] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596: Bootstrap starting.
I20260620 14:47:58.630541 10702 tablet_bootstrap.cc:492] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e: Bootstrap starting.
I20260620 14:47:58.631392 10701 tablet_bootstrap.cc:654] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:58.631507 10703 tablet_bootstrap.cc:492] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806: Bootstrap starting.
I20260620 14:47:58.632794 10701 log.cc:824] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:58.633620 10703 tablet_bootstrap.cc:654] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:58.634457 10703 log.cc:824] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:58.634927 10702 tablet_bootstrap.cc:654] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:58.635141 10701 tablet_bootstrap.cc:492] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596: No bootstrap required, opened a new log
I20260620 14:47:58.635385 10701 ts_tablet_manager.cc:1397] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 14:47:58.635782 10702 log.cc:824] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:58.636610 10703 tablet_bootstrap.cc:492] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806: No bootstrap required, opened a new log
I20260620 14:47:58.636824 10703 ts_tablet_manager.cc:1397] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260620 14:47:58.637835 10702 tablet_bootstrap.cc:492] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e: No bootstrap required, opened a new log
I20260620 14:47:58.638016 10702 ts_tablet_manager.cc:1397] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 14:47:58.639792 10701 raft_consensus.cc:348] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } } peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } }
I20260620 14:47:58.640046 10701 raft_consensus.cc:374] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:58.640100 10701 raft_consensus.cc:729] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab77ffc47683479bb5334a330bb75596, State: Initialized, Role: FOLLOWER
I20260620 14:47:58.640516 10701 consensus_queue.cc:260] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } } peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } }
I20260620 14:47:58.641225 10398 heartbeater.cc:503] Master 127.8.221.254:34971 was elected leader, sending a full tablet report...
I20260620 14:47:58.641160 10703 raft_consensus.cc:348] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } } peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } }
I20260620 14:47:58.641446 10703 raft_consensus.cc:374] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:58.641512 10703 raft_consensus.cc:729] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 051f6d84bf0e4bd0b1cf4aa46f71f806, State: Initialized, Role: FOLLOWER
I20260620 14:47:58.641598 10701 ts_tablet_manager.cc:1428] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:47:58.642019 10703 consensus_queue.cc:260] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } } peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } }
I20260620 14:47:58.642743 10538 heartbeater.cc:503] Master 127.8.221.254:34971 was elected leader, sending a full tablet report...
I20260620 14:47:58.643056 10703 ts_tablet_manager.cc:1428] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260620 14:47:58.643404 10702 raft_consensus.cc:348] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } } peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } }
I20260620 14:47:58.643668 10702 raft_consensus.cc:374] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:58.643728 10702 raft_consensus.cc:729] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68216c4e67024739a5a2539c05322c7e, State: Initialized, Role: FOLLOWER
I20260620 14:47:58.644138 10702 consensus_queue.cc:260] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } } peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } }
I20260620 14:47:58.645002 10678 heartbeater.cc:503] Master 127.8.221.254:34971 was elected leader, sending a full tablet report...
I20260620 14:47:58.645190 10702 ts_tablet_manager.cc:1428] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260620 14:47:58.799652 10679 tablet.cc:2369] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:47:58.808434 10539 tablet.cc:2369] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:47:58.814965 10399 tablet.cc:2369] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:47:58.989184 10709 raft_consensus.cc:482] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:47:58.989413 10709 raft_consensus.cc:504] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } } peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } }
I20260620 14:47:58.990587 10709 leader_election.cc:288] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 051f6d84bf0e4bd0b1cf4aa46f71f806 (127.8.221.194:43339), ab77ffc47683479bb5334a330bb75596 (127.8.221.193:33287)
I20260620 14:47:58.996017 10697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.991226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:45343 (local address 127.8.221.194:43339)
0620 14:47:58.991423 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:58.991429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:58.991452 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:47:58.991691 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:58.991695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:58.991752 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:58.991853 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:58.992395 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.993242 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.994657 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.995269 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.995404 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.995480 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:58.995835 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 14:47:58.995907 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:47:58.996054 10571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.991098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:43339 (local address 127.8.221.195:45343)
0620 14:47:58.991288 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:58.991310 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:58.991499 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:58.991907 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:58.991910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:58.991958 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:58.992176 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.992184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.993357 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.993360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.994524 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.994532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.995424 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.995427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.995614 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.995668 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 14:47:58.995769 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:47:58.995862 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":1,"spinlock_wait_cycles":73088}
I20260620 14:47:58.996869 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.991406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:33287 (local address 127.8.221.195:52297)
0620 14:47:58.991873 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:58.991889 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:58.992057 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:58.992462 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:58.992466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:58.992514 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:58.993065 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.993074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.994168 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.994170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.995395 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:58.995404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.996415 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.996419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:58.996532 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.996561 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 14:47:58.996653 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 14:47:58.996727 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":29,"spinlock_wait_cycles":34944,"thread_start_us":214,"threads_started":1}
I20260620 14:47:58.996965 10700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.991666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:52297 (local address 127.8.221.193:33287)
0620 14:47:58.991847 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:58.991852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:58.991979 (+ 127us) server_negotiation.cc:407] Connection header received
0620 14:47:58.992198 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:58.992202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:58.992265 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:58.992357 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:58.993225 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.994042 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.995541 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.996269 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.996409 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.996463 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:58.996722 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 14:47:58.996829 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:47:58.996953 10487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42abd86bdc974f40a798282888502398" candidate_uuid: "68216c4e67024739a5a2539c05322c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" is_pre_election: true
I20260620 14:47:58.997401 10487 raft_consensus.cc:2454] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68216c4e67024739a5a2539c05322c7e in term 0.
I20260620 14:47:58.997684 10352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42abd86bdc974f40a798282888502398" candidate_uuid: "68216c4e67024739a5a2539c05322c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab77ffc47683479bb5334a330bb75596" is_pre_election: true
I20260620 14:47:58.998083 10352 raft_consensus.cc:2454] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68216c4e67024739a5a2539c05322c7e in term 0.
I20260620 14:47:58.998067 10564 leader_election.cc:302] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 051f6d84bf0e4bd0b1cf4aa46f71f806, 68216c4e67024739a5a2539c05322c7e; no voters:
I20260620 14:47:58.998469 10709 raft_consensus.cc:2781] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:47:58.998555 10709 raft_consensus.cc:482] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:47:58.998586 10709 raft_consensus.cc:3037] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:58.999650 10709 raft_consensus.cc:504] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } } peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } }
I20260620 14:47:59.000018 10709 leader_election.cc:288] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [CANDIDATE]: Term 1 election: Requested vote from peers 051f6d84bf0e4bd0b1cf4aa46f71f806 (127.8.221.194:43339), ab77ffc47683479bb5334a330bb75596 (127.8.221.193:33287)
I20260620 14:47:59.000537 10487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42abd86bdc974f40a798282888502398" candidate_uuid: "68216c4e67024739a5a2539c05322c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806"
I20260620 14:47:59.000664 10487 raft_consensus.cc:3037] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:59.000730 10352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42abd86bdc974f40a798282888502398" candidate_uuid: "68216c4e67024739a5a2539c05322c7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab77ffc47683479bb5334a330bb75596"
I20260620 14:47:59.000933 10352 raft_consensus.cc:3037] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:59.001866 10487 raft_consensus.cc:2454] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68216c4e67024739a5a2539c05322c7e in term 1.
I20260620 14:47:59.002211 10564 leader_election.cc:302] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 051f6d84bf0e4bd0b1cf4aa46f71f806, 68216c4e67024739a5a2539c05322c7e; no voters:
I20260620 14:47:59.002384 10709 raft_consensus.cc:2781] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:59.002573 10709 raft_consensus.cc:686] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [term 1 LEADER]: Becoming Leader. State: Replica: 68216c4e67024739a5a2539c05322c7e, State: Running, Role: LEADER
I20260620 14:47:59.002622 10352 raft_consensus.cc:2454] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68216c4e67024739a5a2539c05322c7e in term 1.
I20260620 14:47:59.002921 10709 consensus_queue.cc:237] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } } peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } }
I20260620 14:47:59.006125 10195 catalog_manager.cc:5495] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 68216c4e67024739a5a2539c05322c7e (127.8.221.195). New cstate: current_term: 1 leader_uuid: "68216c4e67024739a5a2539c05322c7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } health_report { overall_health: UNKNOWN } } }
I20260620 14:47:59.077437 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:59.073195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39792 (local address 127.8.221.195:43197)
0620 14:47:59.073395 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:59.073400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:59.073417 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:47:59.073641 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:59.073646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:59.073719 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:59.073865 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:59.074249 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.074865 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.075808 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.076034 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.076151 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:59.076419 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:59.076919 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:59.077014 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:59.077157 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:47:59.077252 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 14:47:59.084766 10487 raft_consensus.cc:1261] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 1 FOLLOWER]: Refusing update from remote peer 68216c4e67024739a5a2539c05322c7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:59.084820 10352 raft_consensus.cc:1261] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596 [term 1 FOLLOWER]: Refusing update from remote peer 68216c4e67024739a5a2539c05322c7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:59.085561 10709 consensus_queue.cc:1035] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:59.085821 10711 consensus_queue.cc:1035] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:59.092830 10714 mvcc.cc:205] Tried to move back new op lower bound from 7298936336722083840 to 7298936336398995456. Current Snapshot: MvccSnapshot[applied={T|T < 7298936336722083840}]
I20260620 14:47:59.095233 10716 mvcc.cc:205] Tried to move back new op lower bound from 7298936336722083840 to 7298936336398995456. Current Snapshot: MvccSnapshot[applied={T|T < 7298936336722083840}]
I20260620 14:47:59.106666 10192 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:59.106838 10192 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:59.109511 10192 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c: Sending DeleteTablet for 3 replicas of tablet 42abd86bdc974f40a798282888502398
I20260620 14:47:59.110441 10612 tablet_service.cc:1514] Processing DeleteTablet for tablet 42abd86bdc974f40a798282888502398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:59 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:39786
I20260620 14:47:59.110592 10472 tablet_service.cc:1514] Processing DeleteTablet for tablet 42abd86bdc974f40a798282888502398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:59 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:34772
I20260620 14:47:59.110984 10724 tablet_replica.cc:330] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e: stopping tablet replica
I20260620 14:47:59.110931 10332 tablet_service.cc:1514] Processing DeleteTablet for tablet 42abd86bdc974f40a798282888502398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:59 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:40886
I20260620 14:47:59.111284 10724 raft_consensus.cc:2229] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:47:59.111408 10726 tablet_replica.cc:330] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596: stopping tablet replica
I20260620 14:47:59.111374 10192 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:47:59.111727 10724 raft_consensus.cc:2258] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:59.111786 10726 raft_consensus.cc:2229] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 14:47:59.111927 10192 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:47:59.112146 10726 raft_consensus.cc:2258] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:59.113253 10724 ts_tablet_manager.cc:1905] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:59.113472 10725 tablet_replica.cc:330] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806: stopping tablet replica
I20260620 14:47:59.113291 10726 ts_tablet_manager.cc:1905] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:59.113960 10725 raft_consensus.cc:2229] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:59.115024 10725 raft_consensus.cc:2258] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:59.116492 10724 ts_tablet_manager.cc:1918] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:47:59.116585 10724 log.cc:1196] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/42abd86bdc974f40a798282888502398
I20260620 14:47:59.116899 10724 ts_tablet_manager.cc:1939] T 42abd86bdc974f40a798282888502398 P 68216c4e67024739a5a2539c05322c7e: Deleting consensus metadata
I20260620 14:47:59.117969 10725 ts_tablet_manager.cc:1905] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:59.118407 10612 tablet_service.cc:1467] Processing CreateTablet for tablet 74ca1ca4aa334e91878394801a9ad03c (DEFAULT_TABLE table=test-table [id=38770f7b3353451ca8f02d196f49fd5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:59.118752 10612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ca1ca4aa334e91878394801a9ad03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:59.118746 10726 ts_tablet_manager.cc:1918] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:47:59.118849 10726 log.cc:1196] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-0/wal/wals/42abd86bdc974f40a798282888502398
I20260620 14:47:59.119184 10726 ts_tablet_manager.cc:1939] T 42abd86bdc974f40a798282888502398 P ab77ffc47683479bb5334a330bb75596: Deleting consensus metadata
I20260620 14:47:59.120391 10182 catalog_manager.cc:4841] TS 68216c4e67024739a5a2539c05322c7e (127.8.221.195:43197): tablet 42abd86bdc974f40a798282888502398 (table test-table [id=7f0370ef622f475f8b03aba43eaa95fd]) successfully deleted
I20260620 14:47:59.120864 10725 ts_tablet_manager.cc:1918] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:47:59.121025 10725 log.cc:1196] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/42abd86bdc974f40a798282888502398
I20260620 14:47:59.121083 10180 catalog_manager.cc:4841] TS ab77ffc47683479bb5334a330bb75596 (127.8.221.193:33287): tablet 42abd86bdc974f40a798282888502398 (table test-table [id=7f0370ef622f475f8b03aba43eaa95fd]) successfully deleted
I20260620 14:47:59.121465 10702 tablet_bootstrap.cc:492] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e: Bootstrap starting.
I20260620 14:47:59.121479 10725 ts_tablet_manager.cc:1939] T 42abd86bdc974f40a798282888502398 P 051f6d84bf0e4bd0b1cf4aa46f71f806: Deleting consensus metadata
I20260620 14:47:59.121590 10472 tablet_service.cc:1467] Processing CreateTablet for tablet 74ca1ca4aa334e91878394801a9ad03c (DEFAULT_TABLE table=test-table [id=38770f7b3353451ca8f02d196f49fd5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:59.121881 10472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ca1ca4aa334e91878394801a9ad03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:59.122609 10702 tablet_bootstrap.cc:654] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:59.122684 10180 catalog_manager.cc:4841] TS 051f6d84bf0e4bd0b1cf4aa46f71f806 (127.8.221.194:43339): tablet 42abd86bdc974f40a798282888502398 (table test-table [id=7f0370ef622f475f8b03aba43eaa95fd]) successfully deleted
I20260620 14:47:59.123925 10703 tablet_bootstrap.cc:492] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806: Bootstrap starting.
I20260620 14:47:59.123967 10702 tablet_bootstrap.cc:492] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e: No bootstrap required, opened a new log
I20260620 14:47:59.124040 10702 ts_tablet_manager.cc:1397] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:47:59.124488 10332 tablet_service.cc:1467] Processing CreateTablet for tablet 74ca1ca4aa334e91878394801a9ad03c (DEFAULT_TABLE table=test-table [id=38770f7b3353451ca8f02d196f49fd5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:59.124516 10702 raft_consensus.cc:348] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } }
I20260620 14:47:59.124629 10702 raft_consensus.cc:374] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:59.124670 10702 raft_consensus.cc:729] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68216c4e67024739a5a2539c05322c7e, State: Initialized, Role: FOLLOWER
I20260620 14:47:59.124715 10332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74ca1ca4aa334e91878394801a9ad03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:59.124890 10703 tablet_bootstrap.cc:654] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:59.125077 10702 consensus_queue.cc:260] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } }
I20260620 14:47:59.126336 10703 tablet_bootstrap.cc:492] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806: No bootstrap required, opened a new log
I20260620 14:47:59.126406 10703 ts_tablet_manager.cc:1397] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 14:47:59.126806 10703 raft_consensus.cc:348] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } }
I20260620 14:47:59.126915 10703 raft_consensus.cc:374] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:59.126945 10703 raft_consensus.cc:729] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 051f6d84bf0e4bd0b1cf4aa46f71f806, State: Initialized, Role: FOLLOWER
I20260620 14:47:59.127054 10703 consensus_queue.cc:260] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } }
I20260620 14:47:59.127269 10703 ts_tablet_manager.cc:1428] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:47:59.132447 10701 tablet_bootstrap.cc:492] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596: Bootstrap starting.
I20260620 14:47:59.132915 10702 ts_tablet_manager.cc:1428] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20260620 14:47:59.133459 10701 tablet_bootstrap.cc:654] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:59.134961 10701 tablet_bootstrap.cc:492] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596: No bootstrap required, opened a new log
I20260620 14:47:59.135033 10701 ts_tablet_manager.cc:1397] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:47:59.135957 10701 raft_consensus.cc:348] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } }
I20260620 14:47:59.136082 10701 raft_consensus.cc:374] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:59.136117 10701 raft_consensus.cc:729] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab77ffc47683479bb5334a330bb75596, State: Initialized, Role: FOLLOWER
I20260620 14:47:59.136266 10701 consensus_queue.cc:260] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } }
I20260620 14:47:59.136521 10701 ts_tablet_manager.cc:1428] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:47:59.142722 10707 raft_consensus.cc:482] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:47:59.142858 10707 raft_consensus.cc:504] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } }
I20260620 14:47:59.143993 10707 leader_election.cc:288] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 051f6d84bf0e4bd0b1cf4aa46f71f806 (127.8.221.194:43339), 68216c4e67024739a5a2539c05322c7e (127.8.221.195:43197)
I20260620 14:47:59.148391 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:59.144162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:43339 (local address 127.8.221.193:57791)
0620 14:47:59.144567 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:59.144583 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:59.144689 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:59.145090 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:59.145093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:59.145112 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:59.145240 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:59.145248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.146306 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.146309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:59.147274 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:59.147282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.147944 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.147948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:59.148051 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:59.148082 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:47:59.148171 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:47:59.148248 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":100,"threads_started":1}
I20260620 14:47:59.148387 10697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:59.144554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:57791 (local address 127.8.221.194:43339)
0620 14:47:59.144686 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:59.144690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:59.144706 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:47:59.144822 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:59.144826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:59.144871 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:59.144933 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:59.145366 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.146167 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.147405 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.147806 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.147940 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:59.147991 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:59.148186 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 14:47:59.148249 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:47:59.148873 10487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ca1ca4aa334e91878394801a9ad03c" candidate_uuid: "ab77ffc47683479bb5334a330bb75596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" is_pre_election: true
I20260620 14:47:59.149024 10487 raft_consensus.cc:2454] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab77ffc47683479bb5334a330bb75596 in term 0.
I20260620 14:47:59.149407 10284 leader_election.cc:302] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 051f6d84bf0e4bd0b1cf4aa46f71f806, ab77ffc47683479bb5334a330bb75596; no voters:
I20260620 14:47:59.149673 10707 raft_consensus.cc:2781] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:47:59.149729 10707 raft_consensus.cc:482] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:47:59.149768 10707 raft_consensus.cc:3037] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:59.150477 10731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:59.144294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:43197 (local address 127.8.221.193:60189)
0620 14:47:59.144873 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:59.144896 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:59.145043 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:59.145409 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:59.145412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:59.145427 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:59.145574 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:59.145580 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.146819 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.146824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:59.148449 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:59.148462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.149972 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.149976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:59.150092 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:59.150116 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:47:59.150214 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 14:47:59.150296 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":151,"threads_started":1}
I20260620 14:47:59.150511 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:59.144387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:60189 (local address 127.8.221.195:43197)
0620 14:47:59.144720 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:59.144726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:59.144970 (+ 244us) server_negotiation.cc:407] Connection header received
0620 14:47:59.145165 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:59.145168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:59.145215 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:59.145263 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:59.145760 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.146501 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.148676 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.149541 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.149740 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:59.149810 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:59.150273 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 14:47:59.150387 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260620 14:47:59.151130 10707 raft_consensus.cc:504] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } }
I20260620 14:47:59.151189 10632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ca1ca4aa334e91878394801a9ad03c" candidate_uuid: "ab77ffc47683479bb5334a330bb75596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68216c4e67024739a5a2539c05322c7e" is_pre_election: true
I20260620 14:47:59.151412 10632 raft_consensus.cc:2454] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab77ffc47683479bb5334a330bb75596 in term 0.
I20260620 14:47:59.151508 10707 leader_election.cc:288] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [CANDIDATE]: Term 1 election: Requested vote from peers 051f6d84bf0e4bd0b1cf4aa46f71f806 (127.8.221.194:43339), 68216c4e67024739a5a2539c05322c7e (127.8.221.195:43197)
I20260620 14:47:59.151968 10632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ca1ca4aa334e91878394801a9ad03c" candidate_uuid: "ab77ffc47683479bb5334a330bb75596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68216c4e67024739a5a2539c05322c7e"
I20260620 14:47:59.152154 10632 raft_consensus.cc:3037] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:59.152248 10487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74ca1ca4aa334e91878394801a9ad03c" candidate_uuid: "ab77ffc47683479bb5334a330bb75596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806"
I20260620 14:47:59.152387 10487 raft_consensus.cc:3037] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:59.153264 10487 raft_consensus.cc:2454] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab77ffc47683479bb5334a330bb75596 in term 1.
I20260620 14:47:59.153525 10284 leader_election.cc:302] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 051f6d84bf0e4bd0b1cf4aa46f71f806, ab77ffc47683479bb5334a330bb75596; no voters:
I20260620 14:47:59.153643 10632 raft_consensus.cc:2454] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab77ffc47683479bb5334a330bb75596 in term 1.
I20260620 14:47:59.153697 10707 raft_consensus.cc:2781] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:59.153950 10707 raft_consensus.cc:686] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [term 1 LEADER]: Becoming Leader. State: Replica: ab77ffc47683479bb5334a330bb75596, State: Running, Role: LEADER
I20260620 14:47:59.154222 10707 consensus_queue.cc:237] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } }
I20260620 14:47:59.156603 10192 catalog_manager.cc:5495] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab77ffc47683479bb5334a330bb75596 (127.8.221.193). New cstate: current_term: 1 leader_uuid: "ab77ffc47683479bb5334a330bb75596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:47:59 dist-test-slave-65tz krb5kdc[10150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966877, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:47:59.197620 10739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:59.187683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:34971 (local address 127.0.0.1:40018)
0620 14:47:59.187963 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:59.187983 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:59.188083 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:59.188967 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:59.188970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:59.189393 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:59.189473 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:59.189479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.190299 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.190305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:59.190847 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:59.190854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.191285 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.191288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:59.191374 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:59.192117 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:59.192129 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:59.193585 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:59.196012 (+ 2427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:59.196023 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:59.196030 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:59.196269 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:59.196576 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:59.196581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:59.196583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:59.196660 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:59.197032 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:59.197039 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:59.197143 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 14:47:59.197265 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 14:47:59.197420 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":107,"threads_started":1}
I20260620 14:47:59.197619 10740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:59.187969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.8.221.254:34971)
0620 14:47:59.188749 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:59.188755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:59.188767 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:47:59.188797 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:59.188799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:59.188832 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:59.188951 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:59.189586 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.190194 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.190964 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.191177 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.191282 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:59.193728 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:59.193731 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:59.193733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:59.193752 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:59.195866 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:59.196369 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:59.196411 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:59.196413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:59.196483 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:59.196751 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:59.196753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:59.196755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:59.196931 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:59.197032 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:59.197317 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 14:47:59.197476 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":715,"thread_start_us":218,"threads_started":1}
I20260620 14:47:59.201918 10192 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:40018:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:47:59.202690 10192 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:47:59.208258 10472 tablet_service.cc:1467] Processing CreateTablet for tablet 5ef4cdf7417545e88330659e1a47913d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ae336389f2aa473abcbc5ab549f79e9c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:47:59.208258 10332 tablet_service.cc:1467] Processing CreateTablet for tablet 5ef4cdf7417545e88330659e1a47913d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ae336389f2aa473abcbc5ab549f79e9c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:47:59.208635 10472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ef4cdf7417545e88330659e1a47913d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:59.208637 10332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ef4cdf7417545e88330659e1a47913d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:59.208722 10612 tablet_service.cc:1467] Processing CreateTablet for tablet 5ef4cdf7417545e88330659e1a47913d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ae336389f2aa473abcbc5ab549f79e9c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:47:59.209214 10612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ef4cdf7417545e88330659e1a47913d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:59.211580 10703 tablet_bootstrap.cc:492] T 5ef4cdf7417545e88330659e1a47913d P 051f6d84bf0e4bd0b1cf4aa46f71f806: Bootstrap starting.
I20260620 14:47:59.211619 10701 tablet_bootstrap.cc:492] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596: Bootstrap starting.
I20260620 14:47:59.212592 10701 tablet_bootstrap.cc:654] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:59.212592 10703 tablet_bootstrap.cc:654] T 5ef4cdf7417545e88330659e1a47913d P 051f6d84bf0e4bd0b1cf4aa46f71f806: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:59.213728 10702 tablet_bootstrap.cc:492] T 5ef4cdf7417545e88330659e1a47913d P 68216c4e67024739a5a2539c05322c7e: Bootstrap starting.
I20260620 14:47:59.214129 10701 tablet_bootstrap.cc:492] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596: No bootstrap required, opened a new log
I20260620 14:47:59.214213 10701 ts_tablet_manager.cc:1397] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:47:59.214617 10703 tablet_bootstrap.cc:492] T 5ef4cdf7417545e88330659e1a47913d P 051f6d84bf0e4bd0b1cf4aa46f71f806: No bootstrap required, opened a new log
I20260620 14:47:59.214686 10703 ts_tablet_manager.cc:1397] T 5ef4cdf7417545e88330659e1a47913d P 051f6d84bf0e4bd0b1cf4aa46f71f806: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 14:47:59.214684 10701 raft_consensus.cc:348] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } }
I20260620 14:47:59.214800 10701 raft_consensus.cc:374] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:59.214821 10702 tablet_bootstrap.cc:654] T 5ef4cdf7417545e88330659e1a47913d P 68216c4e67024739a5a2539c05322c7e: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:59.214838 10701 raft_consensus.cc:729] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab77ffc47683479bb5334a330bb75596, State: Initialized, Role: FOLLOWER
I20260620 14:47:59.214957 10701 consensus_queue.cc:260] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } }
I20260620 14:47:59.215138 10703 raft_consensus.cc:348] T 5ef4cdf7417545e88330659e1a47913d P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } }
I20260620 14:47:59.215204 10701 ts_tablet_manager.cc:1428] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:47:59.215240 10703 raft_consensus.cc:374] T 5ef4cdf7417545e88330659e1a47913d P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:59.215273 10703 raft_consensus.cc:729] T 5ef4cdf7417545e88330659e1a47913d P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 051f6d84bf0e4bd0b1cf4aa46f71f806, State: Initialized, Role: FOLLOWER
I20260620 14:47:59.215396 10703 consensus_queue.cc:260] T 5ef4cdf7417545e88330659e1a47913d P 051f6d84bf0e4bd0b1cf4aa46f71f806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } }
I20260620 14:47:59.215502 10732 tablet_replica.cc:439] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } } }
I20260620 14:47:59.215636 10732 tablet_replica.cc:442] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:59.215643 10703 ts_tablet_manager.cc:1428] T 5ef4cdf7417545e88330659e1a47913d P 051f6d84bf0e4bd0b1cf4aa46f71f806: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:47:59.216202 10708 tablet_replica.cc:439] T 5ef4cdf7417545e88330659e1a47913d P 051f6d84bf0e4bd0b1cf4aa46f71f806: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } } }
I20260620 14:47:59.216344 10708 tablet_replica.cc:442] T 5ef4cdf7417545e88330659e1a47913d P 051f6d84bf0e4bd0b1cf4aa46f71f806: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:59.218844 10702 tablet_bootstrap.cc:492] T 5ef4cdf7417545e88330659e1a47913d P 68216c4e67024739a5a2539c05322c7e: No bootstrap required, opened a new log
I20260620 14:47:59.218956 10702 ts_tablet_manager.cc:1397] T 5ef4cdf7417545e88330659e1a47913d P 68216c4e67024739a5a2539c05322c7e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260620 14:47:59.219378 10702 raft_consensus.cc:348] T 5ef4cdf7417545e88330659e1a47913d P 68216c4e67024739a5a2539c05322c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } }
I20260620 14:47:59.219480 10702 raft_consensus.cc:374] T 5ef4cdf7417545e88330659e1a47913d P 68216c4e67024739a5a2539c05322c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:59.219501 10702 raft_consensus.cc:729] T 5ef4cdf7417545e88330659e1a47913d P 68216c4e67024739a5a2539c05322c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68216c4e67024739a5a2539c05322c7e, State: Initialized, Role: FOLLOWER
I20260620 14:47:59.219628 10702 consensus_queue.cc:260] T 5ef4cdf7417545e88330659e1a47913d P 68216c4e67024739a5a2539c05322c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } }
I20260620 14:47:59.219848 10702 ts_tablet_manager.cc:1428] T 5ef4cdf7417545e88330659e1a47913d P 68216c4e67024739a5a2539c05322c7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:47:59.219868 10711 tablet_replica.cc:439] T 5ef4cdf7417545e88330659e1a47913d P 68216c4e67024739a5a2539c05322c7e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } } }
I20260620 14:47:59.220036 10711 tablet_replica.cc:442] T 5ef4cdf7417545e88330659e1a47913d P 68216c4e67024739a5a2539c05322c7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:59.307076 10732 raft_consensus.cc:482] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:47:59.307309 10732 raft_consensus.cc:504] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } }
I20260620 14:47:59.307968 10732 leader_election.cc:288] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 051f6d84bf0e4bd0b1cf4aa46f71f806 (127.8.221.194:43339), 68216c4e67024739a5a2539c05322c7e (127.8.221.195:43197)
I20260620 14:47:59.308624 10487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef4cdf7417545e88330659e1a47913d" candidate_uuid: "ab77ffc47683479bb5334a330bb75596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" is_pre_election: true
I20260620 14:47:59.308624 10632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef4cdf7417545e88330659e1a47913d" candidate_uuid: "ab77ffc47683479bb5334a330bb75596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68216c4e67024739a5a2539c05322c7e" is_pre_election: true
I20260620 14:47:59.308797 10487 raft_consensus.cc:2454] T 5ef4cdf7417545e88330659e1a47913d P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab77ffc47683479bb5334a330bb75596 in term 0.
I20260620 14:47:59.308804 10632 raft_consensus.cc:2454] T 5ef4cdf7417545e88330659e1a47913d P 68216c4e67024739a5a2539c05322c7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab77ffc47683479bb5334a330bb75596 in term 0.
I20260620 14:47:59.309266 10284 leader_election.cc:302] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 051f6d84bf0e4bd0b1cf4aa46f71f806, ab77ffc47683479bb5334a330bb75596; no voters:
I20260620 14:47:59.309494 10732 raft_consensus.cc:2781] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:47:59.309554 10732 raft_consensus.cc:482] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:47:59.309592 10732 raft_consensus.cc:3037] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:59.310902 10732 raft_consensus.cc:504] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } }
I20260620 14:47:59.311271 10732 leader_election.cc:288] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 [CANDIDATE]: Term 1 election: Requested vote from peers 051f6d84bf0e4bd0b1cf4aa46f71f806 (127.8.221.194:43339), 68216c4e67024739a5a2539c05322c7e (127.8.221.195:43197)
I20260620 14:47:59.311681 10487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef4cdf7417545e88330659e1a47913d" candidate_uuid: "ab77ffc47683479bb5334a330bb75596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806"
I20260620 14:47:59.311688 10632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ef4cdf7417545e88330659e1a47913d" candidate_uuid: "ab77ffc47683479bb5334a330bb75596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68216c4e67024739a5a2539c05322c7e"
I20260620 14:47:59.311803 10632 raft_consensus.cc:3037] T 5ef4cdf7417545e88330659e1a47913d P 68216c4e67024739a5a2539c05322c7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:59.311810 10487 raft_consensus.cc:3037] T 5ef4cdf7417545e88330659e1a47913d P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:59.312987 10632 raft_consensus.cc:2454] T 5ef4cdf7417545e88330659e1a47913d P 68216c4e67024739a5a2539c05322c7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab77ffc47683479bb5334a330bb75596 in term 1.
I20260620 14:47:59.313315 10487 raft_consensus.cc:2454] T 5ef4cdf7417545e88330659e1a47913d P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab77ffc47683479bb5334a330bb75596 in term 1.
I20260620 14:47:59.313313 10286 leader_election.cc:302] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68216c4e67024739a5a2539c05322c7e, ab77ffc47683479bb5334a330bb75596; no voters:
I20260620 14:47:59.313573 10732 raft_consensus.cc:2781] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:59.313858 10732 raft_consensus.cc:686] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 [term 1 LEADER]: Becoming Leader. State: Replica: ab77ffc47683479bb5334a330bb75596, State: Running, Role: LEADER
I20260620 14:47:59.314028 10732 consensus_queue.cc:237] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } }
I20260620 14:47:59.315062 10707 tablet_replica.cc:439] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596: TxnStatusTablet state changed. Reason: New leader ab77ffc47683479bb5334a330bb75596. Latest consensus state: current_term: 1 leader_uuid: "ab77ffc47683479bb5334a330bb75596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } } }
I20260620 14:47:59.315227 10707 tablet_replica.cc:442] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:47:59.316023 10745 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:47:59.316243 10745 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:47:59.317165 10195 catalog_manager.cc:5495] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab77ffc47683479bb5334a330bb75596 (127.8.221.193). New cstate: current_term: 1 leader_uuid: "ab77ffc47683479bb5334a330bb75596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } health_report { overall_health: UNKNOWN } } }
I20260620 14:47:59.350178 10739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:59.345738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:33287 (local address 127.0.0.1:40898)
0620 14:47:59.345986 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:59.346004 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:59.346121 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:59.346491 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:59.346494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:59.346502 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:47:59.346617 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:59.346623 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.347415 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.347419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:59.348418 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:59.348428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.348901 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.348904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:59.348999 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:59.349051 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:47:59.349835 (+ 784us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:47:59.349851 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 14:47:59.349938 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:47:59.350019 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260620 14:47:59.350205 10700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:59.345910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.8.221.193:33287)
0620 14:47:59.346072 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:59.346076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:59.346091 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:47:59.346289 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:59.346291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:59.346353 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:59.346446 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:59.346726 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.347300 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.348540 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.348778 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.348861 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:59.349194 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:59.349703 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:59.349816 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:59.349995 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 14:47:59.350053 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:47:59.353518 10487 raft_consensus.cc:1261] T 5ef4cdf7417545e88330659e1a47913d P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 1 FOLLOWER]: Refusing update from remote peer ab77ffc47683479bb5334a330bb75596: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:59.353663 10632 raft_consensus.cc:1261] T 5ef4cdf7417545e88330659e1a47913d P 68216c4e67024739a5a2539c05322c7e [term 1 FOLLOWER]: Refusing update from remote peer ab77ffc47683479bb5334a330bb75596: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:59.354089 10732 consensus_queue.cc:1035] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:59.354275 10707 consensus_queue.cc:1035] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:59.355831 10708 tablet_replica.cc:439] T 5ef4cdf7417545e88330659e1a47913d P 051f6d84bf0e4bd0b1cf4aa46f71f806: TxnStatusTablet state changed. Reason: New leader ab77ffc47683479bb5334a330bb75596. Latest consensus state: current_term: 1 leader_uuid: "ab77ffc47683479bb5334a330bb75596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } } }
I20260620 14:47:59.355947 10708 tablet_replica.cc:442] T 5ef4cdf7417545e88330659e1a47913d P 051f6d84bf0e4bd0b1cf4aa46f71f806: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:59.356215 10711 tablet_replica.cc:439] T 5ef4cdf7417545e88330659e1a47913d P 68216c4e67024739a5a2539c05322c7e: TxnStatusTablet state changed. Reason: New leader ab77ffc47683479bb5334a330bb75596. Latest consensus state: current_term: 1 leader_uuid: "ab77ffc47683479bb5334a330bb75596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } } }
I20260620 14:47:59.356335 10711 tablet_replica.cc:442] T 5ef4cdf7417545e88330659e1a47913d P 68216c4e67024739a5a2539c05322c7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:59.358868 10715 mvcc.cc:205] Tried to move back new op lower bound from 7298936337826140160 to 7298936337671991296. Current Snapshot: MvccSnapshot[applied={T|T < 7298936337826140160}]
I20260620 14:47:59.358984 10732 tablet_replica.cc:439] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab77ffc47683479bb5334a330bb75596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } } }
I20260620 14:47:59.359009 10707 tablet_replica.cc:439] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab77ffc47683479bb5334a330bb75596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } } }
I20260620 14:47:59.359107 10732 tablet_replica.cc:442] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:47:59.359126 10707 tablet_replica.cc:442] T 5ef4cdf7417545e88330659e1a47913d P ab77ffc47683479bb5334a330bb75596: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:47:59.360035 10711 tablet_replica.cc:439] T 5ef4cdf7417545e88330659e1a47913d P 68216c4e67024739a5a2539c05322c7e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab77ffc47683479bb5334a330bb75596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } } }
I20260620 14:47:59.360129 10711 tablet_replica.cc:442] T 5ef4cdf7417545e88330659e1a47913d P 68216c4e67024739a5a2539c05322c7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:59.360100 10708 tablet_replica.cc:439] T 5ef4cdf7417545e88330659e1a47913d P 051f6d84bf0e4bd0b1cf4aa46f71f806: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab77ffc47683479bb5334a330bb75596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab77ffc47683479bb5334a330bb75596" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33287 } } peers { permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 } } peers { permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 } } }
I20260620 14:47:59.360206 10708 tablet_replica.cc:442] T 5ef4cdf7417545e88330659e1a47913d P 051f6d84bf0e4bd0b1cf4aa46f71f806: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:59.368227 10700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:59.365179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.8.221.193:33287)
0620 14:47:59.365329 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:59.365334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:59.365347 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:47:59.365392 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:59.365394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:59.365434 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:59.365513 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:59.365772 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.366343 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.367180 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.367417 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.367525 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:59.367788 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:59.367870 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:59.367949 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:59.368038 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 14:47:59.368087 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:47:59.379711 10753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:59.376091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:33287 (local address 127.8.221.193:45203)
0620 14:47:59.376399 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:59.376412 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:59.376539 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:59.376808 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:59.376811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:59.376818 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:47:59.376975 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:59.376984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.377684 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.377690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:59.378506 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:59.378515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.378968 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.378971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:59.379068 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:59.379121 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:47:59.379393 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:47:59.379400 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:47:59.379472 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:47:59.379552 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":21,"thread_start_us":143,"threads_started":1}
I20260620 14:47:59.379873 10700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:59.376161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:45203 (local address 127.8.221.193:33287)
0620 14:47:59.376280 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:59.376285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:59.376488 (+ 203us) server_negotiation.cc:407] Connection header received
0620 14:47:59.376642 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:59.376645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:59.376686 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:59.376800 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:59.377083 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.377584 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.378613 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.378842 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.378959 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:59.379235 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:59.379311 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:59.379403 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:59.379532 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 14:47:59.379624 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":43}
I20260620 14:47:59.387420 10632 raft_consensus.cc:1261] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e [term 1 FOLLOWER]: Refusing update from remote peer ab77ffc47683479bb5334a330bb75596: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:59.387876 10487 raft_consensus.cc:1261] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 1 FOLLOWER]: Refusing update from remote peer ab77ffc47683479bb5334a330bb75596: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:59.387876 10732 consensus_queue.cc:1035] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68216c4e67024739a5a2539c05322c7e" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:59.388334 10732 consensus_queue.cc:1035] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "051f6d84bf0e4bd0b1cf4aa46f71f806" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:59.432662 10192 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:59.432904 10192 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:59.435213 10192 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d70fbd4037324718a68aaf4dcccb973c: Sending DeleteTablet for 3 replicas of tablet 74ca1ca4aa334e91878394801a9ad03c
I20260620 14:47:59.435899 10332 tablet_service.cc:1514] Processing DeleteTablet for tablet 74ca1ca4aa334e91878394801a9ad03c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:59 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:40886
I20260620 14:47:59.435930 10472 tablet_service.cc:1514] Processing DeleteTablet for tablet 74ca1ca4aa334e91878394801a9ad03c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:59 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:34772
I20260620 14:47:59.436087 10612 tablet_service.cc:1514] Processing DeleteTablet for tablet 74ca1ca4aa334e91878394801a9ad03c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:59 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:39786
I20260620 14:47:59.436151 10725 tablet_replica.cc:330] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806: stopping tablet replica
I20260620 14:47:59.436303 10725 raft_consensus.cc:2229] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:59.436331 10726 tablet_replica.cc:330] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596: stopping tablet replica
I20260620 14:47:59.436440 10726 raft_consensus.cc:2229] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:47:59.436465 10725 raft_consensus.cc:2258] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:59.436735 10726 raft_consensus.cc:2258] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:59.436921 10724 tablet_replica.cc:330] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e: stopping tablet replica
I20260620 14:47:59.437044 10724 raft_consensus.cc:2229] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:59.437148 10724 raft_consensus.cc:2258] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:59.437284 10725 ts_tablet_manager.cc:1905] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:59.438144 10726 ts_tablet_manager.cc:1905] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:59.440726 10725 ts_tablet_manager.cc:1918] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:47:59.440785 10725 log.cc:1196] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/74ca1ca4aa334e91878394801a9ad03c
I20260620 14:47:59.440979 10725 ts_tablet_manager.cc:1939] T 74ca1ca4aa334e91878394801a9ad03c P 051f6d84bf0e4bd0b1cf4aa46f71f806: Deleting consensus metadata
I20260620 14:47:59.441411 10726 ts_tablet_manager.cc:1918] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:47:59.441493 10726 log.cc:1196] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-0/wal/wals/74ca1ca4aa334e91878394801a9ad03c
I20260620 14:47:59.441843 10180 catalog_manager.cc:4841] TS 051f6d84bf0e4bd0b1cf4aa46f71f806 (127.8.221.194:43339): tablet 74ca1ca4aa334e91878394801a9ad03c (table test-table [id=38770f7b3353451ca8f02d196f49fd5e]) successfully deleted
I20260620 14:47:59.442071 10726 ts_tablet_manager.cc:1939] T 74ca1ca4aa334e91878394801a9ad03c P ab77ffc47683479bb5334a330bb75596: Deleting consensus metadata
I20260620 14:47:59.442858 10180 catalog_manager.cc:4841] TS ab77ffc47683479bb5334a330bb75596 (127.8.221.193:33287): tablet 74ca1ca4aa334e91878394801a9ad03c (table test-table [id=38770f7b3353451ca8f02d196f49fd5e]) successfully deleted
I20260620 14:47:59.443199 10724 ts_tablet_manager.cc:1905] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:59.445832 10724 ts_tablet_manager.cc:1918] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:47:59.445917 10724 log.cc:1196] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/74ca1ca4aa334e91878394801a9ad03c
I20260620 14:47:59.446179 10724 ts_tablet_manager.cc:1939] T 74ca1ca4aa334e91878394801a9ad03c P 68216c4e67024739a5a2539c05322c7e: Deleting consensus metadata
I20260620 14:47:59.446926 10182 catalog_manager.cc:4841] TS 68216c4e67024739a5a2539c05322c7e (127.8.221.195:43197): tablet 74ca1ca4aa334e91878394801a9ad03c (table test-table [id=38770f7b3353451ca8f02d196f49fd5e]) successfully deleted
Jun 20 14:47:59 dist-test-slave-65tz krb5kdc[10150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966878, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.221.193@KRBTEST.COM
I20260620 14:47:59.453540 10700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:59.441381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40906 (local address 127.8.221.193:33287)
0620 14:47:59.441521 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:59.441526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:59.441977 (+ 451us) server_negotiation.cc:407] Connection header received
0620 14:47:59.442200 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:59.442205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:59.442259 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:59.442355 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:59.443263 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.443816 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.444720 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.444952 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.445059 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:59.449255 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:59.449266 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:59.449282 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:59.449304 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:59.451235 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:59.452079 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:59.452085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:59.452086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:59.452194 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:59.452504 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:59.452506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:59.452508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:59.452774 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:59.452961 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:59.453209 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 14:47:59.453333 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 14:47:59.454201 10392 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40906
I20260620 14:47:59.464221 10740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:59.456132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40020 (local address 127.8.221.254:34971)
0620 14:47:59.456289 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:59.456293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:59.456501 (+ 208us) server_negotiation.cc:407] Connection header received
0620 14:47:59.456662 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:59.456665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:59.456721 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:59.456817 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:59.457448 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.458125 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.458921 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.459187 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.459300 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:59.460631 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:59.460635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:59.460638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:59.460661 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:59.462209 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:59.462658 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:59.462664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:59.462666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:59.462765 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:59.463049 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:59.463052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:59.463054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:59.463244 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:59.463401 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:59.463727 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 14:47:59.463904 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 14:47:59.464704 10192 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40020
I20260620 14:47:59.466355 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 10267
I20260620 14:47:59.476049 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 10407
I20260620 14:47:59.483371 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 10547
I20260620 14:47:59.491565 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 10163
2026-06-20T14:47:59Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3187 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:47:59 dist-test-slave-65tz krb5kdc[10764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:47:59 dist-test-slave-65tz krb5kdc[10764](info): set up 2 sockets
Jun 20 14:47:59 dist-test-slave-65tz krb5kdc[10764](info): commencing operation
krb5kdc: starting...
W20260620 14:48:00.571866 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:48:00 dist-test-slave-65tz krb5kdc[10764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:48:00.701730 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:37987
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:37987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:00.830646 10777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:00.831018 10777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:00.831099 10777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:00.835354 10777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:00.835413 10777 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:48:00.835431 10777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:00.835450 10777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:00.835467 10777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:00.835531 10777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:00.841107 10777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:37987
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:37987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:00.842409 10777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:00.843621 10777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:00.853586 10785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:00.853508 10782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:00.853602 10783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:00.853708 10777 server_base.cc:1034] running on GCE node
I20260620 14:48:00.854514 10777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:00.855798 10777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:00.856977 10777 hybrid_clock.cc:648] HybridClock initialized: now 1781966880856953 us; error 40 us; skew 500 ppm
Jun 20 14:48:00 dist-test-slave-65tz krb5kdc[10764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966880, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:00.860235 10777 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:48:00.861320 10777 webserver.cc:460] Webserver started at http://127.8.221.254:39213/ using document root <none> and password file <none>
I20260620 14:48:00.862052 10777 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:00.862119 10777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:00.862370 10777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:00.864773 10777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "e9b062d0db4646a18e4eb1b55f8fe5a8"
format_stamp: "Formatted at 2026-06-20 14:48:00 on dist-test-slave-65tz"
server_key: "dae192fa36d244bed923d66ac4d215e4"
server_key_iv: "b88f137d480039f73e7107e2ce7fae28"
server_key_version: "encryptionkey@0"
I20260620 14:48:00.865456 10777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "e9b062d0db4646a18e4eb1b55f8fe5a8"
format_stamp: "Formatted at 2026-06-20 14:48:00 on dist-test-slave-65tz"
server_key: "dae192fa36d244bed923d66ac4d215e4"
server_key_iv: "b88f137d480039f73e7107e2ce7fae28"
server_key_version: "encryptionkey@0"
I20260620 14:48:00.870292 10777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:48:00.873287 10792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:00.874797 10777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:00.874960 10777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "e9b062d0db4646a18e4eb1b55f8fe5a8"
format_stamp: "Formatted at 2026-06-20 14:48:00 on dist-test-slave-65tz"
server_key: "dae192fa36d244bed923d66ac4d215e4"
server_key_iv: "b88f137d480039f73e7107e2ce7fae28"
server_key_version: "encryptionkey@0"
I20260620 14:48:00.875142 10777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:00.907029 10777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:00.910925 10777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:00.911164 10777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:00.921133 10777 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:37987
I20260620 14:48:00.921162 10854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:37987 every 8 connection(s)
I20260620 14:48:00.922281 10777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:00.925459 10855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:00.929231 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 10777
I20260620 14:48:00.929342 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:48:00.929684 9079 external_mini_cluster.cc:1428] Setting key f0cbb8d01cf86e94f309fc40eef83fce
I20260620 14:48:00.931653 10855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8: Bootstrap starting.
I20260620 14:48:00.934039 10855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:00.934839 10855 log.cc:824] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:00.936759 10855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8: No bootstrap required, opened a new log
Jun 20 14:48:00 dist-test-slave-65tz krb5kdc[10764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:00.940884 10855 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b062d0db4646a18e4eb1b55f8fe5a8" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 37987 } }
I20260620 14:48:00.941136 10855 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:00.941217 10855 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b062d0db4646a18e4eb1b55f8fe5a8, State: Initialized, Role: FOLLOWER
I20260620 14:48:00.941709 10855 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b062d0db4646a18e4eb1b55f8fe5a8" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 37987 } }
I20260620 14:48:00.941942 10855 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:00.942027 10855 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:00.942133 10855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:00.943396 10855 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b062d0db4646a18e4eb1b55f8fe5a8" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 37987 } }
I20260620 14:48:00.943833 10855 leader_election.cc:302] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9b062d0db4646a18e4eb1b55f8fe5a8; no voters:
I20260620 14:48:00.944211 10855 leader_election.cc:288] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:00.944387 10860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:00.944715 10860 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8 [term 1 LEADER]: Becoming Leader. State: Replica: e9b062d0db4646a18e4eb1b55f8fe5a8, State: Running, Role: LEADER
I20260620 14:48:00.945145 10860 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b062d0db4646a18e4eb1b55f8fe5a8" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 37987 } }
I20260620 14:48:00.945734 10855 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:00.947256 10861 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9b062d0db4646a18e4eb1b55f8fe5a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b062d0db4646a18e4eb1b55f8fe5a8" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 37987 } } }
I20260620 14:48:00.947427 10861 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:00.947777 10862 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9b062d0db4646a18e4eb1b55f8fe5a8. Latest consensus state: current_term: 1 leader_uuid: "e9b062d0db4646a18e4eb1b55f8fe5a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b062d0db4646a18e4eb1b55f8fe5a8" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 37987 } } }
I20260620 14:48:00.947878 10862 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:00.948119 10858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:00.931357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.8.221.254:37987)
0620 14:48:00.931934 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:00.931954 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:00.932040 (+ 86us) server_negotiation.cc:407] Connection header received
0620 14:48:00.932836 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:00.932856 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:00.933270 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:00.933665 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:00.934578 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:00.935712 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:00.936616 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:00.937196 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:00.937390 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:00.940670 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:00.940677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:00.940696 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:00.940730 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:00.943298 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:00.943876 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:00.943884 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:00.943890 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:00.944053 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:00.944454 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:00.944460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:00.944462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:00.945184 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:00.945418 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:00.946417 (+ 999us) server_negotiation.cc:299] Negotiation successful
0620 14:48:00.946813 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":176,"threads_started":1}
I20260620 14:48:00.948489 10869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:00.952548 10869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:00.959472 10869 catalog_manager.cc:1347] Generated new cluster ID: 63f4d45aeee84f69a6fe3ab9acfe23d2
I20260620 14:48:00.959556 10869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:00.970223 10869 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:00.971619 10869 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:00.977466 10869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8: Generated new TSK 0
I20260620 14:48:00.978257 10869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
I20260620 14:48:01.009829 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:37987
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:01.135972 10881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:01.136327 10881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:01.136425 10881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:01.140781 10881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:01.140878 10881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:01.140959 10881 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:48:01.141004 10881 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:48:01.141068 10881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:48:01.147001 10881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.221.254:37987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:01.148463 10881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:01.149684 10881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:01.161123 10889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:01.161124 10887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:01.161497 10881 server_base.cc:1034] running on GCE node
W20260620 14:48:01.161135 10886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:01.162063 10881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:01.162818 10881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:01.163983 10881 hybrid_clock.cc:648] HybridClock initialized: now 1781966881163956 us; error 38 us; skew 500 ppm
Jun 20 14:48:01 dist-test-slave-65tz krb5kdc[10764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966881, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:01.167001 10881 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:48:01.167889 10881 webserver.cc:460] Webserver started at http://127.8.221.193:37317/ using document root <none> and password file <none>
I20260620 14:48:01.168553 10881 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:01.168622 10881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:01.168797 10881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:01.171078 10881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "21bfa42268ed45ab85db0749ee7f9bd6"
format_stamp: "Formatted at 2026-06-20 14:48:01 on dist-test-slave-65tz"
server_key: "5ef9ab5013ff2fd674836cd080a2944b"
server_key_iv: "442162a868e34f5b707276ff656943ec"
server_key_version: "encryptionkey@0"
I20260620 14:48:01.171717 10881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "21bfa42268ed45ab85db0749ee7f9bd6"
format_stamp: "Formatted at 2026-06-20 14:48:01 on dist-test-slave-65tz"
server_key: "5ef9ab5013ff2fd674836cd080a2944b"
server_key_iv: "442162a868e34f5b707276ff656943ec"
server_key_version: "encryptionkey@0"
I20260620 14:48:01.175967 10881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:48:01.178825 10896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:01.180455 10881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:01.180557 10881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "21bfa42268ed45ab85db0749ee7f9bd6"
format_stamp: "Formatted at 2026-06-20 14:48:01 on dist-test-slave-65tz"
server_key: "5ef9ab5013ff2fd674836cd080a2944b"
server_key_iv: "442162a868e34f5b707276ff656943ec"
server_key_version: "encryptionkey@0"
I20260620 14:48:01.180680 10881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:01.191099 10881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:01.194507 10881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:01.194710 10881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:01.196205 10881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:01.196283 10881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:01.196341 10881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:01.196393 10881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:48:01 dist-test-slave-65tz krb5kdc[10764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966881, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:01.208053 10881 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:38463
I20260620 14:48:01.208230 11011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:38463 every 8 connection(s)
I20260620 14:48:01.209087 10881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:01.214519 10858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.197560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:37557 (local address 127.8.221.254:37987)
0620 14:48:01.197744 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:01.197750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:01.198860 (+ 1110us) server_negotiation.cc:407] Connection header received
0620 14:48:01.200493 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:01.200497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:01.200566 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:01.200690 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:01.202557 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.203298 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.205034 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.205294 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.205408 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.208570 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:01.208573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:01.208577 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:01.208606 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:01.212227 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:01.212777 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:01.212780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:01.212781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:01.212866 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:01.213259 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:01.213263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:01.213264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:01.213481 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:01.213584 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:01.214241 (+ 657us) server_negotiation.cc:299] Negotiation successful
0620 14:48:01.214374 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:48:01.215490 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.198159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:37987 (local address 127.8.221.193:37557)
0620 14:48:01.198691 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:01.198747 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:01.200203 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:01.200885 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:01.200895 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:01.201520 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:01.202382 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.202426 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.203457 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.203460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.204914 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.204923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.205455 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.205459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.205668 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.206519 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:01.206540 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:01.208376 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:01.212357 (+ 3981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:01.212363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:01.212376 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:01.212664 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:01.212946 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:01.212948 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:01.212950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:01.213092 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:01.213585 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:01.213590 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:01.213931 (+ 341us) client_negotiation.cc:769] Sending connection context
0620 14:48:01.214104 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 14:48:01.214407 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":153,"threads_started":1}
I20260620 14:48:01.217149 11012 heartbeater.cc:347] Connected to a master server at 127.8.221.254:37987
I20260620 14:48:01.217387 11012 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:01.217559 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 10881
I20260620 14:48:01.217684 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:48:01.218044 9079 external_mini_cluster.cc:1428] Setting key 74d3817a39d505fc5ea946faaa88be61
I20260620 14:48:01.218101 11012 heartbeater.cc:511] Master 127.8.221.254:37987 requested a full tablet report, sending...
I20260620 14:48:01.219743 10808 ts_manager.cc:194] Registered new tserver with Master: 21bfa42268ed45ab85db0749ee7f9bd6 (127.8.221.193:38463)
I20260620 14:48:01.221168 10808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:37557
I20260620 14:48:01.228752 10858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.220189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:50317 (local address 127.8.221.254:37987)
0620 14:48:01.220339 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:01.220345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:01.220673 (+ 328us) server_negotiation.cc:407] Connection header received
0620 14:48:01.220788 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:01.220791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:01.220839 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:01.220925 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:01.221561 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.222356 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.223726 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.223971 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.224067 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.225520 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:01.225523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:01.225525 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:01.225546 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:01.227060 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:01.227579 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:01.227582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:01.227583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:01.227653 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:01.227962 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:01.227964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:01.227965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:01.228112 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:01.228209 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:01.228524 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 14:48:01.228641 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:48:01.229081 11018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.220121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:37987 (local address 127.8.221.193:50317)
0620 14:48:01.220534 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:01.220553 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:01.220680 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:01.220968 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:01.220974 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:01.221249 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:01.221426 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.221436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.222505 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.222511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.223568 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.223581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.224111 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.224117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.224253 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.224836 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:01.224856 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:01.225372 (+ 516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:01.227183 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:01.227188 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:01.227191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:01.227461 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:01.227750 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:01.227755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:01.227759 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:01.227861 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:01.228218 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:01.228221 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:01.228343 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 14:48:01.228469 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 14:48:01.228671 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.8.221.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
I20260620 14:48:01.249375 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:37987
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:01.381496 11021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:01.381868 11021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:01.381960 11021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:01.386289 11021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:01.386404 11021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:01.386492 11021 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:48:01.386539 11021 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:48:01.386603 11021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:48:01.392518 11021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.221.254:37987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:01.394003 11021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:01.395315 11021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:01.406597 11029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:01.406623 11026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:01.406680 11027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:01.406994 11021 server_base.cc:1034] running on GCE node
I20260620 14:48:01.407492 11021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:01.408226 11021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:01.409410 11021 hybrid_clock.cc:648] HybridClock initialized: now 1781966881409392 us; error 39 us; skew 500 ppm
Jun 20 14:48:01 dist-test-slave-65tz krb5kdc[10764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966881, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:01.412477 11021 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:48:01.413440 11021 webserver.cc:460] Webserver started at http://127.8.221.194:41967/ using document root <none> and password file <none>
I20260620 14:48:01.414136 11021 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:01.414206 11021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:01.414426 11021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:01.416671 11021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "b1ac13f44d1643d5ac1ff1307f05b869"
format_stamp: "Formatted at 2026-06-20 14:48:01 on dist-test-slave-65tz"
server_key: "8ac23a74d9f8f012369a1f1d195ae260"
server_key_iv: "6e1a41f1b84ab1a0653c5806b4e5bdf5"
server_key_version: "encryptionkey@0"
I20260620 14:48:01.417259 11021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "b1ac13f44d1643d5ac1ff1307f05b869"
format_stamp: "Formatted at 2026-06-20 14:48:01 on dist-test-slave-65tz"
server_key: "8ac23a74d9f8f012369a1f1d195ae260"
server_key_iv: "6e1a41f1b84ab1a0653c5806b4e5bdf5"
server_key_version: "encryptionkey@0"
I20260620 14:48:01.421531 11021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:48:01.424286 11036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:01.425966 11021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:01.426131 11021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "b1ac13f44d1643d5ac1ff1307f05b869"
format_stamp: "Formatted at 2026-06-20 14:48:01 on dist-test-slave-65tz"
server_key: "8ac23a74d9f8f012369a1f1d195ae260"
server_key_iv: "6e1a41f1b84ab1a0653c5806b4e5bdf5"
server_key_version: "encryptionkey@0"
I20260620 14:48:01.426255 11021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:01.446048 11021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:01.450515 11021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:01.450752 11021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:01.452518 11021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:01.452603 11021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:01.452684 11021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:01.452735 11021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:48:01 dist-test-slave-65tz krb5kdc[10764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966881, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:01.465654 11021 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:42587
I20260620 14:48:01.466663 11151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:42587 every 8 connection(s)
I20260620 14:48:01.467043 11021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:01.467557 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 11021
I20260620 14:48:01.467689 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:01.468025 9079 external_mini_cluster.cc:1428] Setting key a0e8105ef3d2da381cb035373370c84a
I20260620 14:48:01.470084 10858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.453497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:37891 (local address 127.8.221.254:37987)
0620 14:48:01.453709 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:01.453717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:01.454653 (+ 936us) server_negotiation.cc:407] Connection header received
0620 14:48:01.455811 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:01.455814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:01.455871 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:01.455953 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:01.457716 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.458591 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.460448 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.460796 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.460899 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.464539 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:01.464543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:01.464546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:01.464574 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:01.466394 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:01.467056 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:01.467062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:01.467064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:01.467186 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:01.467607 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:01.467611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:01.467612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:01.467860 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:01.467951 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:01.469704 (+ 1753us) server_negotiation.cc:299] Negotiation successful
0620 14:48:01.469891 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:48:01.470712 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.453965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:37987 (local address 127.8.221.194:37891)
0620 14:48:01.454470 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:01.454521 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:01.455575 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:01.456293 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:01.456300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:01.456849 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:01.457557 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.457576 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.458805 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.458809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.460317 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.460330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.460949 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.460953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.461148 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.461959 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:01.461991 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:01.464355 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:01.466550 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:01.466555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:01.466568 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:01.466925 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:01.467284 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:01.467286 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:01.467287 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:01.467469 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:01.468031 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:01.468037 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:01.468386 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 14:48:01.468593 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 14:48:01.468955 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":147,"threads_started":1}
I20260620 14:48:01.472270 11152 heartbeater.cc:347] Connected to a master server at 127.8.221.254:37987
I20260620 14:48:01.472456 11152 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:01.473099 11152 heartbeater.cc:511] Master 127.8.221.254:37987 requested a full tablet report, sending...
I20260620 14:48:01.474351 10808 ts_manager.cc:194] Registered new tserver with Master: b1ac13f44d1643d5ac1ff1307f05b869 (127.8.221.194:42587)
I20260620 14:48:01.475132 10808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:37891
WARNING: no policy specified for kudu/127.8.221.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.195@KRBTEST.COM" created.
I20260620 14:48:01.486142 10858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.474824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:47633 (local address 127.8.221.254:37987)
0620 14:48:01.474977 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:01.474984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:01.478094 (+ 3110us) server_negotiation.cc:407] Connection header received
0620 14:48:01.478216 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:01.478220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:01.478276 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:01.478382 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:01.478928 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.479799 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.480756 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.480961 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.481048 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.482322 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:01.482326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:01.482328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:01.482353 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:01.484106 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:01.484594 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:01.484601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:01.484606 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:01.484700 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:01.485067 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:01.485073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:01.485077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:01.485385 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:01.485473 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:01.485871 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 14:48:01.485989 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:48:01.486238 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.474768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:37987 (local address 127.8.221.194:47633)
0620 14:48:01.477985 (+ 3217us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:01.477998 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:01.478108 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:01.478402 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:01.478404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:01.478643 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:01.478790 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.478796 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.479929 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.479931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.480640 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.480649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.481076 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.481078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.481190 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.481668 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:01.481679 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:01.482137 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:01.484249 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:01.484253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:01.484254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:01.484479 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:01.484802 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:01.484804 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:01.484805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:01.484886 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:01.485493 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:01.485498 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:01.485617 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 14:48:01.485755 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 14:48:01.485986 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3130,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
I20260620 14:48:01.501482 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:37987
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:01.627157 11161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:01.627462 11161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:01.627513 11161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:01.631785 11161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:01.631860 11161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:01.631901 11161 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:48:01.631916 11161 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:48:01.631949 11161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:48:01.637465 11161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.221.254:37987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:01.638777 11161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:01.640002 11161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:01.651265 11166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:01.651266 11167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:01.651266 11169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:01.651775 11161 server_base.cc:1034] running on GCE node
I20260620 14:48:01.652252 11161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:01.652988 11161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:01.654208 11161 hybrid_clock.cc:648] HybridClock initialized: now 1781966881654188 us; error 32 us; skew 500 ppm
Jun 20 14:48:01 dist-test-slave-65tz krb5kdc[10764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966881, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:01.657100 11161 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:48:01.658088 11161 webserver.cc:460] Webserver started at http://127.8.221.195:38853/ using document root <none> and password file <none>
I20260620 14:48:01.658655 11161 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:01.658715 11161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:01.658900 11161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:01.660941 11161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "98876ae946d64ece9558d05acc2d5aad"
format_stamp: "Formatted at 2026-06-20 14:48:01 on dist-test-slave-65tz"
server_key: "566d2bc77f09bada5661f0dfe23a9d1d"
server_key_iv: "3a661456ee048be095f96e9c93caed27"
server_key_version: "encryptionkey@0"
I20260620 14:48:01.661473 11161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "98876ae946d64ece9558d05acc2d5aad"
format_stamp: "Formatted at 2026-06-20 14:48:01 on dist-test-slave-65tz"
server_key: "566d2bc77f09bada5661f0dfe23a9d1d"
server_key_iv: "3a661456ee048be095f96e9c93caed27"
server_key_version: "encryptionkey@0"
I20260620 14:48:01.665541 11161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:48:01.668113 11176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:01.669693 11161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:01.669879 11161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "98876ae946d64ece9558d05acc2d5aad"
format_stamp: "Formatted at 2026-06-20 14:48:01 on dist-test-slave-65tz"
server_key: "566d2bc77f09bada5661f0dfe23a9d1d"
server_key_iv: "3a661456ee048be095f96e9c93caed27"
server_key_version: "encryptionkey@0"
I20260620 14:48:01.670033 11161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:01.683872 11161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:01.687425 11161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:01.687654 11161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:01.689347 11161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:01.689404 11161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:01.689476 11161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:01.689518 11161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:48:01 dist-test-slave-65tz krb5kdc[10764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966881, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:01.702540 11161 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:44945
I20260620 14:48:01.702623 11291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:44945 every 8 connection(s)
I20260620 14:48:01.703609 11161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:01.706188 10858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.690820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:60953 (local address 127.8.221.254:37987)
0620 14:48:01.691005 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:01.691011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:01.692069 (+ 1058us) server_negotiation.cc:407] Connection header received
0620 14:48:01.693773 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:01.693779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:01.693904 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:01.694051 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:01.695838 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.696754 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.698124 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.698472 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.698591 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.701684 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:01.701688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:01.701691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:01.701723 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:01.703532 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:01.704130 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:01.704137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:01.704142 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:01.704227 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:01.704744 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:01.704749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:01.704751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:01.705070 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:01.705199 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:01.705901 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 14:48:01.706051 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 14:48:01.707276 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.691273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:37987 (local address 127.8.221.195:60953)
0620 14:48:01.691878 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:01.691942 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:01.693474 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:01.694284 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:01.694297 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:01.694834 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:01.695643 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.695665 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.696903 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.696907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.697994 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.698006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.698660 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.698664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.698871 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.699683 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:01.699702 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:01.701464 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:01.703661 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:01.703668 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:01.703687 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:01.704010 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:01.704325 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:01.704329 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:01.704331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:01.704552 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:01.705232 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:01.705238 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:01.705539 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 14:48:01.705737 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 14:48:01.706168 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":132,"threads_started":1}
I20260620 14:48:01.708761 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 11161
I20260620 14:48:01.708745 11292 heartbeater.cc:347] Connected to a master server at 127.8.221.254:37987
I20260620 14:48:01.708899 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:01.708952 11292 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:01.709357 9079 external_mini_cluster.cc:1428] Setting key 7c4701ed552390f07c4bdaf5c810b737
I20260620 14:48:01.709590 11292 heartbeater.cc:511] Master 127.8.221.254:37987 requested a full tablet report, sending...
I20260620 14:48:01.711072 10809 ts_manager.cc:194] Registered new tserver with Master: 98876ae946d64ece9558d05acc2d5aad (127.8.221.195:44945)
I20260620 14:48:01.711993 10809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:60953
I20260620 14:48:01.712049 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:48:01.720935 10858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.711770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:42817 (local address 127.8.221.254:37987)
0620 14:48:01.711929 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:01.711934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:01.712160 (+ 226us) server_negotiation.cc:407] Connection header received
0620 14:48:01.712335 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:01.712338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:01.712383 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:01.712474 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:01.712998 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.713760 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.714816 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.715256 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.715395 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.716796 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:01.716800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:01.716802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:01.716832 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:01.718783 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:01.719400 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:01.719406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:01.719420 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:01.719514 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:01.719871 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:01.719877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:01.719882 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:01.720163 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:01.720270 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:01.720679 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 14:48:01.720793 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:48:01.721101 11298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.711754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:37987 (local address 127.8.221.195:42817)
0620 14:48:01.712081 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:01.712097 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:01.712235 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:01.712489 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:01.712492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:01.712739 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:01.712866 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.712873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.713954 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.713957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.714679 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.714688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.715408 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.715411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.715494 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.716216 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:01.716229 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:01.716627 (+ 398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:01.718947 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:01.718952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:01.718954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:01.719271 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:01.719648 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:01.719650 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:01.719652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:01.719755 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:01.720328 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:01.720333 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:01.720474 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 14:48:01.720630 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 14:48:01.720822 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":142,"threads_started":1}
Jun 20 14:48:01 dist-test-slave-65tz krb5kdc[10764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:48:01 dist-test-slave-65tz krb5kdc[10764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:01.744378 10858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.734864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39054 (local address 127.8.221.254:37987)
0620 14:48:01.735043 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:01.735047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:01.735064 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:48:01.735113 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:01.735116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:01.735166 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:01.735259 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:01.736051 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.736584 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.737557 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.737848 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.737960 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.740881 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:01.740885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:01.740888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:01.740920 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:01.742579 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:01.743092 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:01.743095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:01.743097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:01.743187 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:01.743488 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:01.743491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:01.743492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:01.743730 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:01.743864 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:01.744087 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 14:48:01.744215 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 14:48:01.747984 10809 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:48:01.751372 10809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:01.768558 11311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.762090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:44945 (local address 127.0.0.1:34252)
0620 14:48:01.763405 (+ 1315us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:01.763418 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:01.763519 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:01.764246 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:01.764249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:01.764265 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:01.764371 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.764377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.765676 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.765679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.766759 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.766768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.767745 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.767748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.768005 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.768047 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:48:01.768167 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 14:48:01.768364 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1260,"mutex_wait_us":5,"thread_start_us":96,"threads_started":1}
I20260620 14:48:01.768570 11313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.762772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34252 (local address 127.8.221.195:44945)
0620 14:48:01.763245 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:01.763274 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:01.763637 (+ 363us) server_negotiation.cc:407] Connection header received
0620 14:48:01.763721 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:01.763727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:01.763948 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:01.764111 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:01.764507 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.765561 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.766898 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.767598 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.767714 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.767934 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:01.768229 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 14:48:01.768366 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":131,"threads_started":1}
I20260620 14:48:01.769479 11312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.761845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:42587 (local address 127.0.0.1:41172)
0620 14:48:01.763245 (+ 1400us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:01.763262 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:01.763368 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:01.764145 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:01.764151 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:01.764176 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:01.764346 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.764356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.765767 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.765770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.767245 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.767256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.769085 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.769089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.769185 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.769206 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:48:01.769279 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:48:01.769349 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1337,"mutex_wait_us":33,"thread_start_us":150,"threads_started":1}
I20260620 14:48:01.770040 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.762772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.8.221.194:42587)
0620 14:48:01.763291 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:01.763302 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:01.763344 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:48:01.763521 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:01.763528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:01.763805 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:01.763953 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:01.764501 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.765622 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.767412 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.768251 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.768371 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.768506 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:01.769718 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0620 14:48:01.769886 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":133,"threads_started":1}
I20260620 14:48:01.771505 11309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.761758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:38463 (local address 127.0.0.1:47558)
0620 14:48:01.762752 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:01.762809 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:01.762941 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:01.763617 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:01.763624 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:01.763648 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:01.763835 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.763842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.765474 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.765478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.767245 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.767256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.771096 (+ 3840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.771100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.771198 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.771223 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:48:01.771308 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:48:01.771381 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":172,"thread_start_us":135,"threads_started":1}
I20260620 14:48:01.771719 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.761864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47558 (local address 127.8.221.193:38463)
0620 14:48:01.763064 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:01.763074 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:01.763098 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:48:01.763189 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:01.763195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:01.763420 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:01.763595 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:01.764011 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.765340 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.768575 (+ 3235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.770051 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.771064 (+ 1013us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.771205 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:01.771378 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 14:48:01.771504 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1086,"thread_start_us":58,"threads_started":1}
I20260620 14:48:01.772308 11226 tablet_service.cc:1467] Processing CreateTablet for tablet fd33f3170b464e7e91e8e88ff60c52f4 (DEFAULT_TABLE table=test-table [id=fee4e44ed02d421790de7f3b1c9868ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:01.773455 11226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd33f3170b464e7e91e8e88ff60c52f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:01.773352 11086 tablet_service.cc:1467] Processing CreateTablet for tablet fd33f3170b464e7e91e8e88ff60c52f4 (DEFAULT_TABLE table=test-table [id=fee4e44ed02d421790de7f3b1c9868ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:01.774649 11086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd33f3170b464e7e91e8e88ff60c52f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:01.776038 10946 tablet_service.cc:1467] Processing CreateTablet for tablet fd33f3170b464e7e91e8e88ff60c52f4 (DEFAULT_TABLE table=test-table [id=fee4e44ed02d421790de7f3b1c9868ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:01.777098 10946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd33f3170b464e7e91e8e88ff60c52f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:01.781874 11315 tablet_bootstrap.cc:492] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad: Bootstrap starting.
I20260620 14:48:01.783191 11316 tablet_bootstrap.cc:492] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869: Bootstrap starting.
I20260620 14:48:01.785043 11315 tablet_bootstrap.cc:654] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:01.785094 11317 tablet_bootstrap.cc:492] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6: Bootstrap starting.
I20260620 14:48:01.785300 11316 tablet_bootstrap.cc:654] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:01.786311 11316 log.cc:824] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:01.786356 11315 log.cc:824] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:01.787408 11317 tablet_bootstrap.cc:654] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:01.788200 11317 log.cc:824] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:01.789022 11316 tablet_bootstrap.cc:492] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869: No bootstrap required, opened a new log
I20260620 14:48:01.789021 11315 tablet_bootstrap.cc:492] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad: No bootstrap required, opened a new log
I20260620 14:48:01.789261 11316 ts_tablet_manager.cc:1397] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:48:01.789261 11315 ts_tablet_manager.cc:1397] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260620 14:48:01.790462 11317 tablet_bootstrap.cc:492] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6: No bootstrap required, opened a new log
I20260620 14:48:01.790940 11317 ts_tablet_manager.cc:1397] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:48:01.794363 11317 raft_consensus.cc:348] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:01.794625 11317 raft_consensus.cc:374] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:01.794377 11316 raft_consensus.cc:348] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:01.794377 11315 raft_consensus.cc:348] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:01.794699 11317 raft_consensus.cc:729] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21bfa42268ed45ab85db0749ee7f9bd6, State: Initialized, Role: FOLLOWER
I20260620 14:48:01.794711 11316 raft_consensus.cc:374] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:01.794711 11315 raft_consensus.cc:374] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:01.794766 11316 raft_consensus.cc:729] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ac13f44d1643d5ac1ff1307f05b869, State: Initialized, Role: FOLLOWER
I20260620 14:48:01.794766 11315 raft_consensus.cc:729] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98876ae946d64ece9558d05acc2d5aad, State: Initialized, Role: FOLLOWER
I20260620 14:48:01.795221 11317 consensus_queue.cc:260] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:01.795322 11316 consensus_queue.cc:260] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:01.795322 11315 consensus_queue.cc:260] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:01.796165 11292 heartbeater.cc:503] Master 127.8.221.254:37987 was elected leader, sending a full tablet report...
I20260620 14:48:01.796305 11317 ts_tablet_manager.cc:1428] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:48:01.796525 11316 ts_tablet_manager.cc:1428] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 14:48:01.796525 11315 ts_tablet_manager.cc:1428] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 14:48:01.796579 11012 heartbeater.cc:503] Master 127.8.221.254:37987 was elected leader, sending a full tablet report...
I20260620 14:48:01.796926 11152 heartbeater.cc:503] Master 127.8.221.254:37987 was elected leader, sending a full tablet report...
I20260620 14:48:01.861577 11321 raft_consensus.cc:482] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:01.861948 11321 raft_consensus.cc:504] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:01.863585 11321 leader_election.cc:288] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1ac13f44d1643d5ac1ff1307f05b869 (127.8.221.194:42587), 98876ae946d64ece9558d05acc2d5aad (127.8.221.195:44945)
I20260620 14:48:01.870003 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.864025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:36697 (local address 127.8.221.194:42587)
0620 14:48:01.864177 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:01.864183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:01.864583 (+ 400us) server_negotiation.cc:407] Connection header received
0620 14:48:01.864748 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:01.864751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:01.864806 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:01.864975 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:01.865413 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.866428 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.868451 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.869161 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.869288 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.869351 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:01.869719 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 14:48:01.869827 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:48:01.870071 11324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.863862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:42587 (local address 127.8.221.193:36697)
0620 14:48:01.864407 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:01.864423 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:01.864627 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:01.864997 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:01.865000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:01.865012 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:01.865246 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.865251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.866559 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.866563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.868267 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.868277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.869294 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.869298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.869513 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.869596 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 14:48:01.869683 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:48:01.869828 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":347,"spinlock_wait_cycles":30464,"thread_start_us":162,"threads_started":1}
I20260620 14:48:01.870574 11325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.863969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:44945 (local address 127.8.221.193:48071)
0620 14:48:01.864443 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:01.864459 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:01.864577 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:01.864973 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:01.864977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:01.865000 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:01.865238 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.865246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.866559 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.866563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.868264 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.868276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.869294 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.869298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.869531 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.869595 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 14:48:01.869683 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:48:01.869827 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":329,"thread_start_us":190,"threads_started":1}
I20260620 14:48:01.870584 11313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.864092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:48071 (local address 127.8.221.195:44945)
0620 14:48:01.864274 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:01.864280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:01.864581 (+ 301us) server_negotiation.cc:407] Connection header received
0620 14:48:01.864739 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:01.864742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:01.864806 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:01.864917 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:01.865413 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.866405 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.868451 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.869135 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.869288 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.869351 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:01.869719 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 14:48:01.869827 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 14:48:01.870900 11105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd33f3170b464e7e91e8e88ff60c52f4" candidate_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" is_pre_election: true
I20260620 14:48:01.871286 11105 raft_consensus.cc:2454] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21bfa42268ed45ab85db0749ee7f9bd6 in term 0.
I20260620 14:48:01.871363 11246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd33f3170b464e7e91e8e88ff60c52f4" candidate_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98876ae946d64ece9558d05acc2d5aad" is_pre_election: true
I20260620 14:48:01.871774 11246 raft_consensus.cc:2454] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21bfa42268ed45ab85db0749ee7f9bd6 in term 0.
I20260620 14:48:01.871888 10900 leader_election.cc:302] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21bfa42268ed45ab85db0749ee7f9bd6, b1ac13f44d1643d5ac1ff1307f05b869; no voters:
I20260620 14:48:01.872259 11321 raft_consensus.cc:2781] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:01.872445 11321 raft_consensus.cc:482] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:01.872488 11321 raft_consensus.cc:3037] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:01.874017 11321 raft_consensus.cc:504] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:01.874460 11321 leader_election.cc:288] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [CANDIDATE]: Term 1 election: Requested vote from peers b1ac13f44d1643d5ac1ff1307f05b869 (127.8.221.194:42587), 98876ae946d64ece9558d05acc2d5aad (127.8.221.195:44945)
I20260620 14:48:01.874922 11246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd33f3170b464e7e91e8e88ff60c52f4" candidate_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98876ae946d64ece9558d05acc2d5aad"
I20260620 14:48:01.874951 11105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd33f3170b464e7e91e8e88ff60c52f4" candidate_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ac13f44d1643d5ac1ff1307f05b869"
I20260620 14:48:01.875066 11246 raft_consensus.cc:3037] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:01.875082 11105 raft_consensus.cc:3037] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:01.876523 11105 raft_consensus.cc:2454] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21bfa42268ed45ab85db0749ee7f9bd6 in term 1.
I20260620 14:48:01.876523 11246 raft_consensus.cc:2454] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21bfa42268ed45ab85db0749ee7f9bd6 in term 1.
I20260620 14:48:01.876920 10897 leader_election.cc:302] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21bfa42268ed45ab85db0749ee7f9bd6, 98876ae946d64ece9558d05acc2d5aad; no voters:
I20260620 14:48:01.877174 11321 raft_consensus.cc:2781] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:01.877494 11321 raft_consensus.cc:686] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 LEADER]: Becoming Leader. State: Replica: 21bfa42268ed45ab85db0749ee7f9bd6, State: Running, Role: LEADER
I20260620 14:48:01.878044 11321 consensus_queue.cc:237] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:01.882131 10807 catalog_manager.cc:5495] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21bfa42268ed45ab85db0749ee7f9bd6 (127.8.221.193). New cstate: current_term: 1 leader_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:48:01 dist-test-slave-65tz krb5kdc[10764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966880, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:01.945376 11333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.935964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:37987 (local address 127.0.0.1:39066)
0620 14:48:01.936307 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:01.936321 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:01.936519 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:01.936774 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:01.936777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:01.937278 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:01.937375 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.937382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.938242 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.938248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.938811 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:01.938818 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.939277 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.939280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:01.939361 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.940021 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:01.940032 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:01.941826 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:01.943675 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:01.943685 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:01.943691 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:01.943997 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:01.944284 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:01.944287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:01.944288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:01.944387 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:01.944778 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:01.944784 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:01.944923 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 14:48:01.945038 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:48:01.945212 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":135,"threads_started":1}
I20260620 14:48:01.945420 10858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:01.936063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.8.221.254:37987)
0620 14:48:01.936207 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:01.936212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:01.936442 (+ 230us) server_negotiation.cc:407] Connection header received
0620 14:48:01.936623 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:01.936626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:01.936669 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:01.936758 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:01.937499 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.938127 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.938946 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:01.939157 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:01.939264 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:01.942016 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:01.942019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:01.942021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:01.942042 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:01.943531 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:01.944121 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:01.944124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:01.944125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:01.944199 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:01.944477 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:01.944479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:01.944480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:01.944654 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:01.944734 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:01.945095 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 14:48:01.945278 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":50}
I20260620 14:48:01.950459 10807 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:39066:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:48:01.951321 10807 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260620 14:48:01.955504 11293 tablet.cc:2369] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:48:01.957489 11226 tablet_service.cc:1467] Processing CreateTablet for tablet 096313c8f65a4266a011e1454197f07a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d380047554442ccbd4a80f1b5022966]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:48:01.957309 11086 tablet_service.cc:1467] Processing CreateTablet for tablet 096313c8f65a4266a011e1454197f07a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d380047554442ccbd4a80f1b5022966]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:48:01.957363 10946 tablet_service.cc:1467] Processing CreateTablet for tablet 096313c8f65a4266a011e1454197f07a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d380047554442ccbd4a80f1b5022966]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:48:01.958163 11086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096313c8f65a4266a011e1454197f07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:01.958163 11226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096313c8f65a4266a011e1454197f07a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:01.958231 10946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096313c8f65a4266a011e1454197f07a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260620 14:48:01.960876 11013 tablet.cc:2369] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:48:01.961191 11317 tablet_bootstrap.cc:492] T 096313c8f65a4266a011e1454197f07a P 21bfa42268ed45ab85db0749ee7f9bd6: Bootstrap starting.
I20260620 14:48:01.961676 11315 tablet_bootstrap.cc:492] T 096313c8f65a4266a011e1454197f07a P 98876ae946d64ece9558d05acc2d5aad: Bootstrap starting.
I20260620 14:48:01.962412 11316 tablet_bootstrap.cc:492] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869: Bootstrap starting.
I20260620 14:48:01.962734 11317 tablet_bootstrap.cc:654] T 096313c8f65a4266a011e1454197f07a P 21bfa42268ed45ab85db0749ee7f9bd6: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:01.962828 11315 tablet_bootstrap.cc:654] T 096313c8f65a4266a011e1454197f07a P 98876ae946d64ece9558d05acc2d5aad: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:01.964223 11315 tablet_bootstrap.cc:492] T 096313c8f65a4266a011e1454197f07a P 98876ae946d64ece9558d05acc2d5aad: No bootstrap required, opened a new log
I20260620 14:48:01.964332 11315 ts_tablet_manager.cc:1397] T 096313c8f65a4266a011e1454197f07a P 98876ae946d64ece9558d05acc2d5aad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:01.964340 11316 tablet_bootstrap.cc:654] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:01.964341 11317 tablet_bootstrap.cc:492] T 096313c8f65a4266a011e1454197f07a P 21bfa42268ed45ab85db0749ee7f9bd6: No bootstrap required, opened a new log
I20260620 14:48:01.964448 11317 ts_tablet_manager.cc:1397] T 096313c8f65a4266a011e1454197f07a P 21bfa42268ed45ab85db0749ee7f9bd6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:01.964816 11315 raft_consensus.cc:348] T 096313c8f65a4266a011e1454197f07a P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:01.964947 11315 raft_consensus.cc:374] T 096313c8f65a4266a011e1454197f07a P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:01.965000 11315 raft_consensus.cc:729] T 096313c8f65a4266a011e1454197f07a P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98876ae946d64ece9558d05acc2d5aad, State: Initialized, Role: FOLLOWER
I20260620 14:48:01.965037 11317 raft_consensus.cc:348] T 096313c8f65a4266a011e1454197f07a P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:01.965173 11317 raft_consensus.cc:374] T 096313c8f65a4266a011e1454197f07a P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:01.965215 11317 raft_consensus.cc:729] T 096313c8f65a4266a011e1454197f07a P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21bfa42268ed45ab85db0749ee7f9bd6, State: Initialized, Role: FOLLOWER
I20260620 14:48:01.965160 11315 consensus_queue.cc:260] T 096313c8f65a4266a011e1454197f07a P 98876ae946d64ece9558d05acc2d5aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:01.965379 11317 consensus_queue.cc:260] T 096313c8f65a4266a011e1454197f07a P 21bfa42268ed45ab85db0749ee7f9bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:01.965476 11323 tablet_replica.cc:439] T 096313c8f65a4266a011e1454197f07a P 98876ae946d64ece9558d05acc2d5aad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } } }
I20260620 14:48:01.965643 11323 tablet_replica.cc:442] T 096313c8f65a4266a011e1454197f07a P 98876ae946d64ece9558d05acc2d5aad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:01.965667 11317 ts_tablet_manager.cc:1428] T 096313c8f65a4266a011e1454197f07a P 21bfa42268ed45ab85db0749ee7f9bd6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 14:48:01.965955 11315 ts_tablet_manager.cc:1428] T 096313c8f65a4266a011e1454197f07a P 98876ae946d64ece9558d05acc2d5aad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:48:01.965919 11321 tablet_replica.cc:439] T 096313c8f65a4266a011e1454197f07a P 21bfa42268ed45ab85db0749ee7f9bd6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } } }
I20260620 14:48:01.966046 11321 tablet_replica.cc:442] T 096313c8f65a4266a011e1454197f07a P 21bfa42268ed45ab85db0749ee7f9bd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:01.966490 11316 tablet_bootstrap.cc:492] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869: No bootstrap required, opened a new log
I20260620 14:48:01.966638 11316 ts_tablet_manager.cc:1397] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 14:48:01.967226 11316 raft_consensus.cc:348] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:01.967427 11316 raft_consensus.cc:374] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:01.967486 11316 raft_consensus.cc:729] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ac13f44d1643d5ac1ff1307f05b869, State: Initialized, Role: FOLLOWER
I20260620 14:48:01.967679 11316 consensus_queue.cc:260] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:01.967952 11322 tablet_replica.cc:439] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } } }
I20260620 14:48:01.968103 11322 tablet_replica.cc:442] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:01.968341 11316 ts_tablet_manager.cc:1428] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260620 14:48:01.968433 11153 tablet.cc:2369] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:48:02.020462 11322 raft_consensus.cc:482] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:02.020666 11322 raft_consensus.cc:504] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:02.021994 11322 leader_election.cc:288] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21bfa42268ed45ab85db0749ee7f9bd6 (127.8.221.193:38463), 98876ae946d64ece9558d05acc2d5aad (127.8.221.195:44945)
I20260620 14:48:02.026875 11313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:02.022505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:59743 (local address 127.8.221.195:44945)
0620 14:48:02.022670 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:02.022676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:02.023035 (+ 359us) server_negotiation.cc:407] Connection header received
0620 14:48:02.023228 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:02.023231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:02.023292 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:02.023385 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:02.023742 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.024368 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.025666 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.026191 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.026338 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:02.026403 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:02.026636 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 14:48:02.026712 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 14:48:02.026883 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:02.022437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:44945 (local address 127.8.221.194:59743)
0620 14:48:02.022933 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:02.022948 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:02.023096 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:02.023399 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:02.023402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:02.023423 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:02.023602 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:02.023609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.024493 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.024500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:02.025520 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:02.025529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.026337 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.026341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:02.026460 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:02.026489 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 14:48:02.026618 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 14:48:02.026712 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":2,"thread_start_us":102,"threads_started":1}
I20260620 14:48:02.027055 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:02.022511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:55547 (local address 127.8.221.193:38463)
0620 14:48:02.022667 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:02.022675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:02.022945 (+ 270us) server_negotiation.cc:407] Connection header received
0620 14:48:02.023138 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:02.023146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:02.023193 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:02.023284 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:02.023584 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.024297 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.025665 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.026184 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.026320 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:02.026404 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:02.026763 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 14:48:02.026858 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:48:02.027269 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:02.022406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:38463 (local address 127.8.221.194:55547)
0620 14:48:02.022821 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:02.022835 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:02.023003 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:02.023298 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:02.023301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:02.023318 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:02.023472 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:02.023478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.024423 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.024427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:02.025531 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:02.025541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.026314 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.026318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:02.026425 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:02.026609 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 14:48:02.026743 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 14:48:02.026889 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":88,"thread_start_us":98,"threads_started":1}
I20260620 14:48:02.027716 11246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096313c8f65a4266a011e1454197f07a" candidate_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98876ae946d64ece9558d05acc2d5aad" is_pre_election: true
I20260620 14:48:02.027944 11246 raft_consensus.cc:2454] T 096313c8f65a4266a011e1454197f07a P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1ac13f44d1643d5ac1ff1307f05b869 in term 0.
I20260620 14:48:02.027981 10966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096313c8f65a4266a011e1454197f07a" candidate_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" is_pre_election: true
I20260620 14:48:02.028285 10966 raft_consensus.cc:2454] T 096313c8f65a4266a011e1454197f07a P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1ac13f44d1643d5ac1ff1307f05b869 in term 0.
I20260620 14:48:02.028491 11037 leader_election.cc:302] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98876ae946d64ece9558d05acc2d5aad, b1ac13f44d1643d5ac1ff1307f05b869; no voters:
I20260620 14:48:02.028891 11322 raft_consensus.cc:2781] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:02.028968 11322 raft_consensus.cc:482] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:02.029023 11322 raft_consensus.cc:3037] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:02.030392 11322 raft_consensus.cc:504] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:02.030871 11322 leader_election.cc:288] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 [CANDIDATE]: Term 1 election: Requested vote from peers 21bfa42268ed45ab85db0749ee7f9bd6 (127.8.221.193:38463), 98876ae946d64ece9558d05acc2d5aad (127.8.221.195:44945)
I20260620 14:48:02.031347 11246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096313c8f65a4266a011e1454197f07a" candidate_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98876ae946d64ece9558d05acc2d5aad"
I20260620 14:48:02.031347 10966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "096313c8f65a4266a011e1454197f07a" candidate_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21bfa42268ed45ab85db0749ee7f9bd6"
I20260620 14:48:02.031517 10966 raft_consensus.cc:3037] T 096313c8f65a4266a011e1454197f07a P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:02.031517 11246 raft_consensus.cc:3037] T 096313c8f65a4266a011e1454197f07a P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:02.032997 10966 raft_consensus.cc:2454] T 096313c8f65a4266a011e1454197f07a P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1ac13f44d1643d5ac1ff1307f05b869 in term 1.
I20260620 14:48:02.032997 11246 raft_consensus.cc:2454] T 096313c8f65a4266a011e1454197f07a P 98876ae946d64ece9558d05acc2d5aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1ac13f44d1643d5ac1ff1307f05b869 in term 1.
I20260620 14:48:02.033406 11037 leader_election.cc:302] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98876ae946d64ece9558d05acc2d5aad, b1ac13f44d1643d5ac1ff1307f05b869; no voters:
I20260620 14:48:02.033601 11322 raft_consensus.cc:2781] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:02.033928 11322 raft_consensus.cc:686] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 [term 1 LEADER]: Becoming Leader. State: Replica: b1ac13f44d1643d5ac1ff1307f05b869, State: Running, Role: LEADER
I20260620 14:48:02.034406 11322 consensus_queue.cc:237] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:02.037081 11339 tablet_replica.cc:439] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869: TxnStatusTablet state changed. Reason: New leader b1ac13f44d1643d5ac1ff1307f05b869. Latest consensus state: current_term: 1 leader_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } } }
I20260620 14:48:02.037248 11339 tablet_replica.cc:442] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:48:02.037585 11341 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:48:02.037858 11341 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:48:02.038245 10808 catalog_manager.cc:5495] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1ac13f44d1643d5ac1ff1307f05b869 (127.8.221.194). New cstate: current_term: 1 leader_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } health_report { overall_health: UNKNOWN } } }
I20260620 14:48:02.058672 11333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:02.053602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:42587 (local address 127.0.0.1:41184)
0620 14:48:02.053782 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:02.053827 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:02.053944 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:02.054426 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:02.054429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:02.054437 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:48:02.054551 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:02.054556 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.055726 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.055729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:02.056417 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:02.056427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.057060 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.057063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:02.057185 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:02.057241 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:48:02.058279 (+ 1038us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:48:02.058297 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:48:02.058378 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:48:02.058457 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 14:48:02.058676 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:02.053742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41184 (local address 127.8.221.194:42587)
0620 14:48:02.053921 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:02.053927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:02.053949 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:48:02.054138 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:02.054142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:02.054223 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:02.054359 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:02.054687 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.055572 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.056558 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.056916 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.057028 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:02.057417 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:02.058154 (+ 737us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:02.058285 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:02.058422 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 14:48:02.058495 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:48:02.065861 10966 raft_consensus.cc:1261] T 096313c8f65a4266a011e1454197f07a P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 FOLLOWER]: Refusing update from remote peer b1ac13f44d1643d5ac1ff1307f05b869: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:02.065961 11246 raft_consensus.cc:1261] T 096313c8f65a4266a011e1454197f07a P 98876ae946d64ece9558d05acc2d5aad [term 1 FOLLOWER]: Refusing update from remote peer b1ac13f44d1643d5ac1ff1307f05b869: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:02.066716 11339 consensus_queue.cc:1035] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:02.067013 11322 consensus_queue.cc:1035] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:02.071439 11321 tablet_replica.cc:439] T 096313c8f65a4266a011e1454197f07a P 21bfa42268ed45ab85db0749ee7f9bd6: TxnStatusTablet state changed. Reason: New leader b1ac13f44d1643d5ac1ff1307f05b869. Latest consensus state: current_term: 1 leader_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } } }
I20260620 14:48:02.071595 11321 tablet_replica.cc:442] T 096313c8f65a4266a011e1454197f07a P 21bfa42268ed45ab85db0749ee7f9bd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:02.072738 11323 tablet_replica.cc:439] T 096313c8f65a4266a011e1454197f07a P 98876ae946d64ece9558d05acc2d5aad: TxnStatusTablet state changed. Reason: New leader b1ac13f44d1643d5ac1ff1307f05b869. Latest consensus state: current_term: 1 leader_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } } }
I20260620 14:48:02.072887 11323 tablet_replica.cc:442] T 096313c8f65a4266a011e1454197f07a P 98876ae946d64ece9558d05acc2d5aad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:02.076973 11322 tablet_replica.cc:439] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } } }
I20260620 14:48:02.077157 11322 tablet_replica.cc:442] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:48:02.077313 11322 tablet_replica.cc:439] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } } }
I20260620 14:48:02.077414 11322 tablet_replica.cc:442] T 096313c8f65a4266a011e1454197f07a P b1ac13f44d1643d5ac1ff1307f05b869: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:48:02.077571 11323 tablet_replica.cc:439] T 096313c8f65a4266a011e1454197f07a P 98876ae946d64ece9558d05acc2d5aad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } } }
I20260620 14:48:02.077697 11323 tablet_replica.cc:442] T 096313c8f65a4266a011e1454197f07a P 98876ae946d64ece9558d05acc2d5aad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:02.079504 11346 mvcc.cc:205] Tried to move back new op lower bound from 7298936348932665344 to 7298936348816101376. Current Snapshot: MvccSnapshot[applied={T|T < 7298936348932665344}]
I20260620 14:48:02.079717 11321 tablet_replica.cc:439] T 096313c8f65a4266a011e1454197f07a P 21bfa42268ed45ab85db0749ee7f9bd6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } } }
I20260620 14:48:02.079835 11321 tablet_replica.cc:442] T 096313c8f65a4266a011e1454197f07a P 21bfa42268ed45ab85db0749ee7f9bd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:02.079775 11345 mvcc.cc:205] Tried to move back new op lower bound from 7298936348932665344 to 7298936348816101376. Current Snapshot: MvccSnapshot[applied={T|T < 7298936348932665344}]
I20260620 14:48:02.091140 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:02.086349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47568 (local address 127.8.221.193:38463)
0620 14:48:02.086493 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:02.086497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:02.086515 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:48:02.086702 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:02.086706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:02.086791 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:02.086964 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:02.087280 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.088127 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.089083 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.089451 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.089570 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:02.089896 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:02.090634 (+ 738us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:02.090761 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:02.090877 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 14:48:02.090959 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:48:02.102799 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:02.099149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:42587 (local address 127.8.221.193:52671)
0620 14:48:02.099547 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:02.099561 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:02.099713 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:02.099972 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:02.099975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:02.099981 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:48:02.100114 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:02.100120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.100838 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.100844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:02.101503 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:02.101511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.102013 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.102016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:02.102117 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:02.102158 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:48:02.102487 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:48:02.102496 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 14:48:02.102558 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:48:02.102634 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":161,"threads_started":1}
I20260620 14:48:02.102782 11314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:02.099287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:52671 (local address 127.8.221.194:42587)
0620 14:48:02.099427 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:02.099432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:02.099642 (+ 210us) server_negotiation.cc:407] Connection header received
0620 14:48:02.099825 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:02.099830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:02.099880 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:02.099961 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:02.100222 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.100722 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.101620 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.101903 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.102010 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:02.102323 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:02.102401 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:02.102483 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:02.102576 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 14:48:02.102624 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:48:02.111023 11246 raft_consensus.cc:1261] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad [term 1 FOLLOWER]: Refusing update from remote peer 21bfa42268ed45ab85db0749ee7f9bd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:02.111378 11105 raft_consensus.cc:1261] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869 [term 1 FOLLOWER]: Refusing update from remote peer 21bfa42268ed45ab85db0749ee7f9bd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:02.111624 11321 consensus_queue.cc:1035] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:02.111876 11326 consensus_queue.cc:1035] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:02.138464 11310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:02.132597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:54349 (local address 127.8.221.193:38463)
0620 14:48:02.132751 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:02.132757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:02.134210 (+ 1453us) server_negotiation.cc:407] Connection header received
0620 14:48:02.134417 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:02.134420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:02.134481 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:02.134575 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:02.134884 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.135724 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.137009 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.137362 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.137470 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:02.137968 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:02.138060 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:02.138135 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:02.138268 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 14:48:02.138342 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:48:02.138829 11361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:02.132501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:38463 (local address 127.8.221.194:54349)
0620 14:48:02.134095 (+ 1594us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:02.134114 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:02.134277 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:02.134629 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:02.134632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:02.134651 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:48:02.134766 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:02.134771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.135846 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.135849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:02.136796 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:02.136805 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.137561 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.137565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:02.137677 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:02.137745 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:48:02.138161 (+ 416us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:48:02.138168 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:48:02.138242 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:48:02.138344 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1437,"thread_start_us":167,"threads_started":1}
I20260620 14:48:02.166656 10807 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:02.166939 10807 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:02.170558 10807 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8: Sending DeleteTablet for 3 replicas of tablet fd33f3170b464e7e91e8e88ff60c52f4
I20260620 14:48:02.171566 11086 tablet_service.cc:1514] Processing DeleteTablet for tablet fd33f3170b464e7e91e8e88ff60c52f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:02 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:41172
I20260620 14:48:02.171787 10946 tablet_service.cc:1514] Processing DeleteTablet for tablet fd33f3170b464e7e91e8e88ff60c52f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:02 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:47558
I20260620 14:48:02.172262 11364 tablet_replica.cc:330] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869: stopping tablet replica
I20260620 14:48:02.172719 11364 raft_consensus.cc:2229] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:02.172712 11226 tablet_service.cc:1514] Processing DeleteTablet for tablet fd33f3170b464e7e91e8e88ff60c52f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:02 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:34252
I20260620 14:48:02.173022 11364 raft_consensus.cc:2258] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:02.173534 11365 tablet_replica.cc:330] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6: stopping tablet replica
I20260620 14:48:02.173816 11366 tablet_replica.cc:330] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad: stopping tablet replica
I20260620 14:48:02.174291 10807 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:48:02.174860 10807 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:02.175772 11364 ts_tablet_manager.cc:1905] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:02.177011 11366 raft_consensus.cc:2229] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:02.177285 11366 raft_consensus.cc:2258] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:02.177845 11365 raft_consensus.cc:2229] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:02.179824 11365 raft_consensus.cc:2258] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:02.180583 11364 ts_tablet_manager.cc:1918] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:48:02.180754 11364 log.cc:1196] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/fd33f3170b464e7e91e8e88ff60c52f4
I20260620 14:48:02.181147 11364 ts_tablet_manager.cc:1939] T fd33f3170b464e7e91e8e88ff60c52f4 P b1ac13f44d1643d5ac1ff1307f05b869: Deleting consensus metadata
I20260620 14:48:02.181552 11086 tablet_service.cc:1467] Processing CreateTablet for tablet 209eff792cad44628d1847c474c55ee5 (DEFAULT_TABLE table=test-table [id=25b608ed8d5a41bf855a8f24fb7cf68d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:02.181628 10946 tablet_service.cc:1467] Processing CreateTablet for tablet 209eff792cad44628d1847c474c55ee5 (DEFAULT_TABLE table=test-table [id=25b608ed8d5a41bf855a8f24fb7cf68d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:02.181967 11086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209eff792cad44628d1847c474c55ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:02.182108 11226 tablet_service.cc:1467] Processing CreateTablet for tablet 209eff792cad44628d1847c474c55ee5 (DEFAULT_TABLE table=test-table [id=25b608ed8d5a41bf855a8f24fb7cf68d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:02.182399 11226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209eff792cad44628d1847c474c55ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:02.182912 10796 catalog_manager.cc:4841] TS b1ac13f44d1643d5ac1ff1307f05b869 (127.8.221.194:42587): tablet fd33f3170b464e7e91e8e88ff60c52f4 (table test-table [id=fee4e44ed02d421790de7f3b1c9868ab]) successfully deleted
I20260620 14:48:02.185115 11316 tablet_bootstrap.cc:492] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869: Bootstrap starting.
I20260620 14:48:02.185925 10946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 209eff792cad44628d1847c474c55ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:02.186245 11316 tablet_bootstrap.cc:654] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:02.187867 11316 tablet_bootstrap.cc:492] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869: No bootstrap required, opened a new log
I20260620 14:48:02.187947 11316 ts_tablet_manager.cc:1397] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:02.188393 11316 raft_consensus.cc:348] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:02.188505 11316 raft_consensus.cc:374] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:02.188535 11316 raft_consensus.cc:729] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ac13f44d1643d5ac1ff1307f05b869, State: Initialized, Role: FOLLOWER
I20260620 14:48:02.188665 11316 consensus_queue.cc:260] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:02.188910 11316 ts_tablet_manager.cc:1428] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:02.188905 11315 tablet_bootstrap.cc:492] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad: Bootstrap starting.
I20260620 14:48:02.190138 11315 tablet_bootstrap.cc:654] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:02.191356 11317 tablet_bootstrap.cc:492] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6: Bootstrap starting.
I20260620 14:48:02.192601 11317 tablet_bootstrap.cc:654] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:02.194447 11317 tablet_bootstrap.cc:492] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6: No bootstrap required, opened a new log
I20260620 14:48:02.194556 11317 ts_tablet_manager.cc:1397] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:02.195254 11317 raft_consensus.cc:348] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:02.195433 11317 raft_consensus.cc:374] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:02.195475 11317 raft_consensus.cc:729] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21bfa42268ed45ab85db0749ee7f9bd6, State: Initialized, Role: FOLLOWER
I20260620 14:48:02.195660 11317 consensus_queue.cc:260] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:02.196134 11317 ts_tablet_manager.cc:1428] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:48:02.197598 11315 tablet_bootstrap.cc:492] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad: No bootstrap required, opened a new log
I20260620 14:48:02.197714 11315 ts_tablet_manager.cc:1397] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260620 14:48:02.198551 11315 raft_consensus.cc:348] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:02.198697 11315 raft_consensus.cc:374] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:02.198740 11315 raft_consensus.cc:729] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98876ae946d64ece9558d05acc2d5aad, State: Initialized, Role: FOLLOWER
I20260620 14:48:02.199451 11315 consensus_queue.cc:260] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:02.200624 11315 ts_tablet_manager.cc:1428] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:48:02.202926 11366 ts_tablet_manager.cc:1905] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:02.207113 11366 ts_tablet_manager.cc:1918] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:48:02.207230 11366 log.cc:1196] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/fd33f3170b464e7e91e8e88ff60c52f4
I20260620 14:48:02.209730 11365 ts_tablet_manager.cc:1905] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:02.213680 11365 ts_tablet_manager.cc:1918] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:48:02.213788 11365 log.cc:1196] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/wal/wals/fd33f3170b464e7e91e8e88ff60c52f4
I20260620 14:48:02.214214 11365 ts_tablet_manager.cc:1939] T fd33f3170b464e7e91e8e88ff60c52f4 P 21bfa42268ed45ab85db0749ee7f9bd6: Deleting consensus metadata
I20260620 14:48:02.215821 10794 catalog_manager.cc:4841] TS 21bfa42268ed45ab85db0749ee7f9bd6 (127.8.221.193:38463): tablet fd33f3170b464e7e91e8e88ff60c52f4 (table test-table [id=fee4e44ed02d421790de7f3b1c9868ab]) successfully deleted
I20260620 14:48:02.216581 11366 ts_tablet_manager.cc:1939] T fd33f3170b464e7e91e8e88ff60c52f4 P 98876ae946d64ece9558d05acc2d5aad: Deleting consensus metadata
I20260620 14:48:02.217885 11321 raft_consensus.cc:482] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:02.218011 11321 raft_consensus.cc:504] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:02.218526 11321 leader_election.cc:288] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1ac13f44d1643d5ac1ff1307f05b869 (127.8.221.194:42587), 98876ae946d64ece9558d05acc2d5aad (127.8.221.195:44945)
I20260620 14:48:02.219444 11105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209eff792cad44628d1847c474c55ee5" candidate_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" is_pre_election: true
I20260620 14:48:02.219627 11105 raft_consensus.cc:2454] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21bfa42268ed45ab85db0749ee7f9bd6 in term 0.
I20260620 14:48:02.219910 11246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209eff792cad44628d1847c474c55ee5" candidate_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98876ae946d64ece9558d05acc2d5aad" is_pre_election: true
I20260620 14:48:02.220075 11246 raft_consensus.cc:2454] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21bfa42268ed45ab85db0749ee7f9bd6 in term 0.
I20260620 14:48:02.220463 10897 leader_election.cc:302] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21bfa42268ed45ab85db0749ee7f9bd6, 98876ae946d64ece9558d05acc2d5aad; no voters:
I20260620 14:48:02.220950 11321 raft_consensus.cc:2781] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:02.221027 11321 raft_consensus.cc:482] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:02.221083 11321 raft_consensus.cc:3037] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:02.223007 11321 raft_consensus.cc:504] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:02.223433 11321 leader_election.cc:288] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [CANDIDATE]: Term 1 election: Requested vote from peers b1ac13f44d1643d5ac1ff1307f05b869 (127.8.221.194:42587), 98876ae946d64ece9558d05acc2d5aad (127.8.221.195:44945)
I20260620 14:48:02.224213 11246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209eff792cad44628d1847c474c55ee5" candidate_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98876ae946d64ece9558d05acc2d5aad"
I20260620 14:48:02.224363 11246 raft_consensus.cc:3037] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:02.225690 11246 raft_consensus.cc:2454] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21bfa42268ed45ab85db0749ee7f9bd6 in term 1.
I20260620 14:48:02.226087 11105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "209eff792cad44628d1847c474c55ee5" candidate_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ac13f44d1643d5ac1ff1307f05b869"
I20260620 14:48:02.226269 11105 raft_consensus.cc:3037] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:02.227526 11105 raft_consensus.cc:2454] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21bfa42268ed45ab85db0749ee7f9bd6 in term 1.
I20260620 14:48:02.227756 10897 leader_election.cc:302] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21bfa42268ed45ab85db0749ee7f9bd6, 98876ae946d64ece9558d05acc2d5aad; no voters:
I20260620 14:48:02.228195 11321 raft_consensus.cc:2781] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:02.228614 11321 raft_consensus.cc:686] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 LEADER]: Becoming Leader. State: Replica: 21bfa42268ed45ab85db0749ee7f9bd6, State: Running, Role: LEADER
I20260620 14:48:02.228783 11321 consensus_queue.cc:237] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:02.230852 10807 catalog_manager.cc:5495] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21bfa42268ed45ab85db0749ee7f9bd6 (127.8.221.193). New cstate: current_term: 1 leader_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } health_report { overall_health: UNKNOWN } } }
I20260620 14:48:02.233626 10793 catalog_manager.cc:4841] TS 98876ae946d64ece9558d05acc2d5aad (127.8.221.195:44945): tablet fd33f3170b464e7e91e8e88ff60c52f4 (table test-table [id=fee4e44ed02d421790de7f3b1c9868ab]) successfully deleted
I20260620 14:48:02.255156 11246 raft_consensus.cc:1261] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad [term 1 FOLLOWER]: Refusing update from remote peer 21bfa42268ed45ab85db0749ee7f9bd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:02.255313 11105 raft_consensus.cc:1261] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869 [term 1 FOLLOWER]: Refusing update from remote peer 21bfa42268ed45ab85db0749ee7f9bd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:02.255712 11326 consensus_queue.cc:1035] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:02.256021 11326 consensus_queue.cc:1035] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:02.267040 11344 mvcc.cc:205] Tried to move back new op lower bound from 7298936349711024128 to 7298936349610950656. Current Snapshot: MvccSnapshot[applied={T|T < 7298936349711024128}]
W20260620 14:48:02.278693 11346 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:48:03.848341 10808 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:03.848518 10808 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:03.851604 10808 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8: Sending DeleteTablet for 3 replicas of tablet 209eff792cad44628d1847c474c55ee5
I20260620 14:48:03.852227 10946 tablet_service.cc:1514] Processing DeleteTablet for tablet 209eff792cad44628d1847c474c55ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:03 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:47558
I20260620 14:48:03.852284 11226 tablet_service.cc:1514] Processing DeleteTablet for tablet 209eff792cad44628d1847c474c55ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:03 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:34252
I20260620 14:48:03.852368 11086 tablet_service.cc:1514] Processing DeleteTablet for tablet 209eff792cad44628d1847c474c55ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:03 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:41172
I20260620 14:48:03.852671 11394 tablet_replica.cc:330] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad: stopping tablet replica
I20260620 14:48:03.852753 11393 tablet_replica.cc:330] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6: stopping tablet replica
I20260620 14:48:03.852766 11395 tablet_replica.cc:330] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869: stopping tablet replica
I20260620 14:48:03.852901 11393 raft_consensus.cc:2229] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:03.852911 11395 raft_consensus.cc:2229] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:03.852912 11394 raft_consensus.cc:2229] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:03.853057 11395 raft_consensus.cc:2258] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:03.853129 11393 raft_consensus.cc:2258] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:03.853235 11394 raft_consensus.cc:2258] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:03.854043 11395 ts_tablet_manager.cc:1905] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:03.854267 11393 ts_tablet_manager.cc:1905] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:03.857301 11395 ts_tablet_manager.cc:1918] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:48:03.857422 11395 log.cc:1196] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/209eff792cad44628d1847c474c55ee5
I20260620 14:48:03.857610 11393 ts_tablet_manager.cc:1918] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:48:03.857721 11393 log.cc:1196] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/wal/wals/209eff792cad44628d1847c474c55ee5
I20260620 14:48:03.857771 11395 ts_tablet_manager.cc:1939] T 209eff792cad44628d1847c474c55ee5 P b1ac13f44d1643d5ac1ff1307f05b869: Deleting consensus metadata
I20260620 14:48:03.858090 11393 ts_tablet_manager.cc:1939] T 209eff792cad44628d1847c474c55ee5 P 21bfa42268ed45ab85db0749ee7f9bd6: Deleting consensus metadata
I20260620 14:48:03.858780 10796 catalog_manager.cc:4841] TS b1ac13f44d1643d5ac1ff1307f05b869 (127.8.221.194:42587): tablet 209eff792cad44628d1847c474c55ee5 (table test-table [id=25b608ed8d5a41bf855a8f24fb7cf68d]) successfully deleted
I20260620 14:48:03.859195 11394 ts_tablet_manager.cc:1905] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:03.859176 10794 catalog_manager.cc:4841] TS 21bfa42268ed45ab85db0749ee7f9bd6 (127.8.221.193:38463): tablet 209eff792cad44628d1847c474c55ee5 (table test-table [id=25b608ed8d5a41bf855a8f24fb7cf68d]) successfully deleted
I20260620 14:48:03.862143 11394 ts_tablet_manager.cc:1918] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:48:03.862272 11394 log.cc:1196] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/209eff792cad44628d1847c474c55ee5
I20260620 14:48:03.862648 11394 ts_tablet_manager.cc:1939] T 209eff792cad44628d1847c474c55ee5 P 98876ae946d64ece9558d05acc2d5aad: Deleting consensus metadata
I20260620 14:48:03.863718 10793 catalog_manager.cc:4841] TS 98876ae946d64ece9558d05acc2d5aad (127.8.221.195:44945): tablet 209eff792cad44628d1847c474c55ee5 (table test-table [id=25b608ed8d5a41bf855a8f24fb7cf68d]) successfully deleted
Jun 20 14:48:03 dist-test-slave-65tz krb5kdc[10764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966883, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:48:03 dist-test-slave-65tz krb5kdc[10764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966883, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:03.891520 11403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:03.876155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.8.221.254:37987)
0620 14:48:03.876545 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:03.876553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:03.876572 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:48:03.876678 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:03.876682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:03.876734 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:03.876826 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:03.877615 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:03.878270 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:03.879302 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:03.879577 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:03.879700 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:03.888262 (+ 8562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:03.888266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:03.888269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:03.888299 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:03.889762 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:03.890284 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:03.890287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:03.890288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:03.890364 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:03.890622 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:03.890624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:03.890625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:03.890808 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:03.890919 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:03.891272 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 14:48:03.891380 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":159,"threads_started":1}
I20260620 14:48:03.894975 10808 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:48:03.895586 10808 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:03.901738 10946 tablet_service.cc:1467] Processing CreateTablet for tablet e623899346284250a6457717ed3cbb4f (DEFAULT_TABLE table=test-table [id=7dbee02cd48f4621baee6d8999bcd874]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:03.901897 11086 tablet_service.cc:1467] Processing CreateTablet for tablet e623899346284250a6457717ed3cbb4f (DEFAULT_TABLE table=test-table [id=7dbee02cd48f4621baee6d8999bcd874]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:03.902020 11226 tablet_service.cc:1467] Processing CreateTablet for tablet e623899346284250a6457717ed3cbb4f (DEFAULT_TABLE table=test-table [id=7dbee02cd48f4621baee6d8999bcd874]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:03.902108 10946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e623899346284250a6457717ed3cbb4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:03.902305 11226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e623899346284250a6457717ed3cbb4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:03.902307 11086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e623899346284250a6457717ed3cbb4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:03.905169 11405 tablet_bootstrap.cc:492] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad: Bootstrap starting.
I20260620 14:48:03.905171 11406 tablet_bootstrap.cc:492] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869: Bootstrap starting.
I20260620 14:48:03.905527 11404 tablet_bootstrap.cc:492] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6: Bootstrap starting.
I20260620 14:48:03.906139 11405 tablet_bootstrap.cc:654] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:03.906139 11406 tablet_bootstrap.cc:654] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:03.906630 11404 tablet_bootstrap.cc:654] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:03.907684 11406 tablet_bootstrap.cc:492] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869: No bootstrap required, opened a new log
I20260620 14:48:03.907820 11406 ts_tablet_manager.cc:1397] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:03.908075 11405 tablet_bootstrap.cc:492] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad: No bootstrap required, opened a new log
I20260620 14:48:03.908200 11405 ts_tablet_manager.cc:1397] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:03.908313 11406 raft_consensus.cc:348] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:03.908450 11406 raft_consensus.cc:374] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:03.908493 11406 raft_consensus.cc:729] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ac13f44d1643d5ac1ff1307f05b869, State: Initialized, Role: FOLLOWER
I20260620 14:48:03.908586 11404 tablet_bootstrap.cc:492] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6: No bootstrap required, opened a new log
I20260620 14:48:03.908670 11406 consensus_queue.cc:260] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:03.908740 11404 ts_tablet_manager.cc:1397] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:03.908859 11405 raft_consensus.cc:348] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:03.908972 11406 ts_tablet_manager.cc:1428] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:03.909005 11405 raft_consensus.cc:374] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:03.909050 11405 raft_consensus.cc:729] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98876ae946d64ece9558d05acc2d5aad, State: Initialized, Role: FOLLOWER
I20260620 14:48:03.909212 11405 consensus_queue.cc:260] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:03.909672 11405 ts_tablet_manager.cc:1428] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 14:48:03.909595 11404 raft_consensus.cc:348] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:03.909780 11404 raft_consensus.cc:374] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:03.909878 11404 raft_consensus.cc:729] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21bfa42268ed45ab85db0749ee7f9bd6, State: Initialized, Role: FOLLOWER
I20260620 14:48:03.910095 11404 consensus_queue.cc:260] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:03.910413 11404 ts_tablet_manager.cc:1428] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:48:03.966387 11410 raft_consensus.cc:482] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:03.966560 11410 raft_consensus.cc:504] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:03.967914 11410 leader_election.cc:288] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21bfa42268ed45ab85db0749ee7f9bd6 (127.8.221.193:38463), b1ac13f44d1643d5ac1ff1307f05b869 (127.8.221.194:42587)
I20260620 14:48:03.973062 11414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:03.968465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:57609 (local address 127.8.221.194:42587)
0620 14:48:03.968793 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:03.968806 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:03.968911 (+ 105us) server_negotiation.cc:407] Connection header received
0620 14:48:03.969146 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:03.969191 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:03.969260 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:03.969398 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:03.969869 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:03.970599 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:03.971751 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:03.972347 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:03.972557 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:03.972636 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:03.972780 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 14:48:03.972885 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":120,"threads_started":1}
I20260620 14:48:03.973079 11413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:03.968329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:42587 (local address 127.8.221.195:57609)
0620 14:48:03.968783 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:03.968801 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:03.968971 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:03.969468 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:03.969471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:03.969489 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:03.969677 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:03.969685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:03.970762 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:03.970765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:03.971623 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:03.971631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:03.972492 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:03.972496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:03.972615 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:03.972662 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:48:03.972749 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:48:03.972819 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":3,"thread_start_us":127,"threads_started":1}
I20260620 14:48:03.973124 11412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:03.968288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:56637 (local address 127.8.221.193:38463)
0620 14:48:03.968731 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:03.968742 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:03.968765 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:48:03.969003 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:03.969010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:03.969075 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:03.969176 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:03.969639 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:03.970276 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:03.971804 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:03.972373 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:03.972626 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:03.972699 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:03.972914 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 14:48:03.972995 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":196,"threads_started":1}
I20260620 14:48:03.973131 11411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:03.968158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:38463 (local address 127.8.221.195:56637)
0620 14:48:03.968660 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:03.968680 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:03.968824 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:03.969240 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:03.969243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:03.969266 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:03.969474 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:03.969485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:03.970400 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:03.970404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:03.971602 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:03.971619 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:03.972619 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:03.972623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:03.972768 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:03.972798 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 14:48:03.972882 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:48:03.972967 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":163,"threads_started":1}
I20260620 14:48:03.973917 10966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e623899346284250a6457717ed3cbb4f" candidate_uuid: "98876ae946d64ece9558d05acc2d5aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" is_pre_election: true
I20260620 14:48:03.974040 11105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e623899346284250a6457717ed3cbb4f" candidate_uuid: "98876ae946d64ece9558d05acc2d5aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" is_pre_election: true
I20260620 14:48:03.974102 10966 raft_consensus.cc:2454] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98876ae946d64ece9558d05acc2d5aad in term 0.
I20260620 14:48:03.974242 11105 raft_consensus.cc:2454] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98876ae946d64ece9558d05acc2d5aad in term 0.
I20260620 14:48:03.974607 11178 leader_election.cc:302] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21bfa42268ed45ab85db0749ee7f9bd6, 98876ae946d64ece9558d05acc2d5aad, b1ac13f44d1643d5ac1ff1307f05b869; no voters:
I20260620 14:48:03.974994 11410 raft_consensus.cc:2781] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:03.975072 11410 raft_consensus.cc:482] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:03.975116 11410 raft_consensus.cc:3037] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:03.976449 11410 raft_consensus.cc:504] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:03.976845 11410 leader_election.cc:288] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [CANDIDATE]: Term 1 election: Requested vote from peers 21bfa42268ed45ab85db0749ee7f9bd6 (127.8.221.193:38463), b1ac13f44d1643d5ac1ff1307f05b869 (127.8.221.194:42587)
I20260620 14:48:03.977169 10966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e623899346284250a6457717ed3cbb4f" candidate_uuid: "98876ae946d64ece9558d05acc2d5aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21bfa42268ed45ab85db0749ee7f9bd6"
I20260620 14:48:03.977245 11105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e623899346284250a6457717ed3cbb4f" candidate_uuid: "98876ae946d64ece9558d05acc2d5aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ac13f44d1643d5ac1ff1307f05b869"
I20260620 14:48:03.977304 10966 raft_consensus.cc:3037] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:03.977340 11105 raft_consensus.cc:3037] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:03.978595 11105 raft_consensus.cc:2454] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98876ae946d64ece9558d05acc2d5aad in term 1.
I20260620 14:48:03.978595 10966 raft_consensus.cc:2454] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98876ae946d64ece9558d05acc2d5aad in term 1.
I20260620 14:48:03.979020 11180 leader_election.cc:302] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98876ae946d64ece9558d05acc2d5aad, b1ac13f44d1643d5ac1ff1307f05b869; no voters:
I20260620 14:48:03.979288 11410 raft_consensus.cc:2781] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:03.979558 11410 raft_consensus.cc:686] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [term 1 LEADER]: Becoming Leader. State: Replica: 98876ae946d64ece9558d05acc2d5aad, State: Running, Role: LEADER
I20260620 14:48:03.979919 11410 consensus_queue.cc:237] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:03.982703 10807 catalog_manager.cc:5495] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad reported cstate change: term changed from 0 to 1, leader changed from <none> to 98876ae946d64ece9558d05acc2d5aad (127.8.221.195). New cstate: current_term: 1 leader_uuid: "98876ae946d64ece9558d05acc2d5aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } health_report { overall_health: HEALTHY } } }
I20260620 14:48:04.017315 11419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.012932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.8.221.195:44945)
0620 14:48:04.013229 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:04.013236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:04.013250 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:48:04.013344 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:04.013346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:04.013408 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:04.013534 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:04.013871 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.014456 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.015351 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.015602 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.015684 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.016026 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:04.016780 (+ 754us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:04.016889 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:04.017045 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 14:48:04.017101 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":129,"threads_started":1}
I20260620 14:48:04.029650 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.025862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:42587 (local address 127.8.221.195:36273)
0620 14:48:04.026277 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:04.026312 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:04.026450 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:04.026726 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:04.026729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:04.026737 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:48:04.026875 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:04.026882 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.027672 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.027678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:04.028340 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:04.028348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.028814 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.028818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:04.028913 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.028970 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:48:04.029282 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:48:04.029298 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 14:48:04.029359 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:48:04.029442 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":164,"threads_started":1}
I20260620 14:48:04.029623 11414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.026069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:36273 (local address 127.8.221.194:42587)
0620 14:48:04.026247 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:04.026253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:04.026390 (+ 137us) server_negotiation.cc:407] Connection header received
0620 14:48:04.026569 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:04.026572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:04.026630 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:04.026718 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:04.026995 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.027561 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.028462 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.028689 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.028814 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.029115 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:04.029198 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:04.029279 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:04.029379 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 14:48:04.029434 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:48:04.036543 10966 raft_consensus.cc:1261] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 FOLLOWER]: Refusing update from remote peer 98876ae946d64ece9558d05acc2d5aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:04.036706 11105 raft_consensus.cc:1261] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869 [term 1 FOLLOWER]: Refusing update from remote peer 98876ae946d64ece9558d05acc2d5aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:04.037129 11410 consensus_queue.cc:1035] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:04.037356 11415 consensus_queue.cc:1035] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:04.057016 11427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.053070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:44945 (local address 127.8.221.194:53433)
0620 14:48:04.053416 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:04.053431 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:04.053537 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:04.053744 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:04.053746 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:04.053752 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:48:04.053933 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:04.053938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.054694 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.054698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:04.055569 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:04.055578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.056212 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.056218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:04.056323 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.056354 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:48:04.056672 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:48:04.056679 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:48:04.056759 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:48:04.056854 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":141,"threads_started":1}
I20260620 14:48:04.057016 11419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.053123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:53433 (local address 127.8.221.195:44945)
0620 14:48:04.053254 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:04.053258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:04.053521 (+ 263us) server_negotiation.cc:407] Connection header received
0620 14:48:04.053619 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:04.053622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:04.053667 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:04.053738 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:04.054062 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.054584 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.055693 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.056057 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.056171 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.056491 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:04.056580 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:04.056656 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:04.056780 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 14:48:04.056850 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:48:04.083357 10808 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:04.083602 10808 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:04.086398 10808 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8: Sending DeleteTablet for 3 replicas of tablet e623899346284250a6457717ed3cbb4f
I20260620 14:48:04.087072 10946 tablet_service.cc:1514] Processing DeleteTablet for tablet e623899346284250a6457717ed3cbb4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:04 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:47558
I20260620 14:48:04.087273 11393 tablet_replica.cc:330] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6: stopping tablet replica
I20260620 14:48:04.087253 11086 tablet_service.cc:1514] Processing DeleteTablet for tablet e623899346284250a6457717ed3cbb4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:04 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:41172
I20260620 14:48:04.087397 11393 raft_consensus.cc:2229] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:04.087510 11395 tablet_replica.cc:330] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869: stopping tablet replica
I20260620 14:48:04.087518 11393 raft_consensus.cc:2258] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:04.087656 11395 raft_consensus.cc:2229] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:04.087680 11226 tablet_service.cc:1514] Processing DeleteTablet for tablet e623899346284250a6457717ed3cbb4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:04 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:34252
I20260620 14:48:04.087805 11395 raft_consensus.cc:2258] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:04.087981 11394 tablet_replica.cc:330] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad: stopping tablet replica
I20260620 14:48:04.088121 11394 raft_consensus.cc:2229] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:04.088091 10808 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:48:04.088455 11394 raft_consensus.cc:2258] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 14:48:04.088531 10808 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:04.088670 11395 ts_tablet_manager.cc:1905] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:04.089689 11393 ts_tablet_manager.cc:1905] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:04.091747 11395 ts_tablet_manager.cc:1918] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:48:04.091832 11395 log.cc:1196] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/e623899346284250a6457717ed3cbb4f
I20260620 14:48:04.092118 11395 ts_tablet_manager.cc:1939] T e623899346284250a6457717ed3cbb4f P b1ac13f44d1643d5ac1ff1307f05b869: Deleting consensus metadata
I20260620 14:48:04.092489 11393 ts_tablet_manager.cc:1918] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:48:04.092557 11393 log.cc:1196] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-0/wal/wals/e623899346284250a6457717ed3cbb4f
I20260620 14:48:04.092820 11393 ts_tablet_manager.cc:1939] T e623899346284250a6457717ed3cbb4f P 21bfa42268ed45ab85db0749ee7f9bd6: Deleting consensus metadata
I20260620 14:48:04.092998 10796 catalog_manager.cc:4841] TS b1ac13f44d1643d5ac1ff1307f05b869 (127.8.221.194:42587): tablet e623899346284250a6457717ed3cbb4f (table test-table [id=7dbee02cd48f4621baee6d8999bcd874]) successfully deleted
I20260620 14:48:04.093456 10946 tablet_service.cc:1467] Processing CreateTablet for tablet ff85d28489a6462aa681ff3f00d567de (DEFAULT_TABLE table=test-table [id=56c330dc47324014afbd7a6dc8f6acf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:04.093529 11086 tablet_service.cc:1467] Processing CreateTablet for tablet ff85d28489a6462aa681ff3f00d567de (DEFAULT_TABLE table=test-table [id=56c330dc47324014afbd7a6dc8f6acf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:04.093719 10946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff85d28489a6462aa681ff3f00d567de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:04.094532 11086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff85d28489a6462aa681ff3f00d567de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:04.095448 11404 tablet_bootstrap.cc:492] T ff85d28489a6462aa681ff3f00d567de P 21bfa42268ed45ab85db0749ee7f9bd6: Bootstrap starting.
I20260620 14:48:04.095676 10794 catalog_manager.cc:4841] TS 21bfa42268ed45ab85db0749ee7f9bd6 (127.8.221.193:38463): tablet e623899346284250a6457717ed3cbb4f (table test-table [id=7dbee02cd48f4621baee6d8999bcd874]) successfully deleted
I20260620 14:48:04.096379 11404 tablet_bootstrap.cc:654] T ff85d28489a6462aa681ff3f00d567de P 21bfa42268ed45ab85db0749ee7f9bd6: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:04.096764 11226 tablet_service.cc:1467] Processing CreateTablet for tablet ff85d28489a6462aa681ff3f00d567de (DEFAULT_TABLE table=test-table [id=56c330dc47324014afbd7a6dc8f6acf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:04.097034 11226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff85d28489a6462aa681ff3f00d567de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:04.097302 11394 ts_tablet_manager.cc:1905] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:04.099383 11394 ts_tablet_manager.cc:1918] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:48:04.099460 11394 log.cc:1196] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/e623899346284250a6457717ed3cbb4f
I20260620 14:48:04.099555 11405 tablet_bootstrap.cc:492] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad: Bootstrap starting.
I20260620 14:48:04.099639 11406 tablet_bootstrap.cc:492] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869: Bootstrap starting.
I20260620 14:48:04.099715 11394 ts_tablet_manager.cc:1939] T e623899346284250a6457717ed3cbb4f P 98876ae946d64ece9558d05acc2d5aad: Deleting consensus metadata
I20260620 14:48:04.100258 11404 tablet_bootstrap.cc:492] T ff85d28489a6462aa681ff3f00d567de P 21bfa42268ed45ab85db0749ee7f9bd6: No bootstrap required, opened a new log
I20260620 14:48:04.100337 11404 ts_tablet_manager.cc:1397] T ff85d28489a6462aa681ff3f00d567de P 21bfa42268ed45ab85db0749ee7f9bd6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260620 14:48:04.100561 11406 tablet_bootstrap.cc:654] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:04.100678 11405 tablet_bootstrap.cc:654] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:04.100790 11404 raft_consensus.cc:348] T ff85d28489a6462aa681ff3f00d567de P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:04.100914 11404 raft_consensus.cc:374] T ff85d28489a6462aa681ff3f00d567de P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:04.100947 11404 raft_consensus.cc:729] T ff85d28489a6462aa681ff3f00d567de P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21bfa42268ed45ab85db0749ee7f9bd6, State: Initialized, Role: FOLLOWER
I20260620 14:48:04.101094 11404 consensus_queue.cc:260] T ff85d28489a6462aa681ff3f00d567de P 21bfa42268ed45ab85db0749ee7f9bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:04.101424 11404 ts_tablet_manager.cc:1428] T ff85d28489a6462aa681ff3f00d567de P 21bfa42268ed45ab85db0749ee7f9bd6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260620 14:48:04.102311 10793 catalog_manager.cc:4841] TS 98876ae946d64ece9558d05acc2d5aad (127.8.221.195:44945): tablet e623899346284250a6457717ed3cbb4f (table test-table [id=7dbee02cd48f4621baee6d8999bcd874]) successfully deleted
I20260620 14:48:04.103062 11406 tablet_bootstrap.cc:492] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869: No bootstrap required, opened a new log
I20260620 14:48:04.103124 11406 ts_tablet_manager.cc:1397] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 14:48:04.103588 11406 raft_consensus.cc:348] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:04.103726 11406 raft_consensus.cc:374] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:04.103765 11406 raft_consensus.cc:729] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ac13f44d1643d5ac1ff1307f05b869, State: Initialized, Role: FOLLOWER
I20260620 14:48:04.103921 11406 consensus_queue.cc:260] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:04.104229 11406 ts_tablet_manager.cc:1428] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:48:04.104911 11405 tablet_bootstrap.cc:492] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad: No bootstrap required, opened a new log
I20260620 14:48:04.105026 11405 ts_tablet_manager.cc:1397] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 14:48:04.105425 11405 raft_consensus.cc:348] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:04.105561 11405 raft_consensus.cc:374] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:04.105602 11405 raft_consensus.cc:729] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98876ae946d64ece9558d05acc2d5aad, State: Initialized, Role: FOLLOWER
I20260620 14:48:04.105751 11405 consensus_queue.cc:260] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:04.106050 11405 ts_tablet_manager.cc:1428] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:04.107721 11425 raft_consensus.cc:482] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:04.107816 11425 raft_consensus.cc:504] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:04.108506 10966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff85d28489a6462aa681ff3f00d567de" candidate_uuid: "98876ae946d64ece9558d05acc2d5aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" is_pre_election: true
I20260620 14:48:04.108635 11105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff85d28489a6462aa681ff3f00d567de" candidate_uuid: "98876ae946d64ece9558d05acc2d5aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" is_pre_election: true
I20260620 14:48:04.108650 10966 raft_consensus.cc:2454] T ff85d28489a6462aa681ff3f00d567de P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98876ae946d64ece9558d05acc2d5aad in term 0.
I20260620 14:48:04.108747 11105 raft_consensus.cc:2454] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98876ae946d64ece9558d05acc2d5aad in term 0.
I20260620 14:48:04.108958 11425 leader_election.cc:288] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21bfa42268ed45ab85db0749ee7f9bd6 (127.8.221.193:38463), b1ac13f44d1643d5ac1ff1307f05b869 (127.8.221.194:42587)
I20260620 14:48:04.109005 11180 leader_election.cc:302] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21bfa42268ed45ab85db0749ee7f9bd6, 98876ae946d64ece9558d05acc2d5aad, b1ac13f44d1643d5ac1ff1307f05b869; no voters:
I20260620 14:48:04.109225 11425 raft_consensus.cc:2781] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:04.109293 11425 raft_consensus.cc:482] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:04.109344 11425 raft_consensus.cc:3037] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:04.110743 11425 raft_consensus.cc:504] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:04.111124 11425 leader_election.cc:288] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [CANDIDATE]: Term 1 election: Requested vote from peers 21bfa42268ed45ab85db0749ee7f9bd6 (127.8.221.193:38463), b1ac13f44d1643d5ac1ff1307f05b869 (127.8.221.194:42587)
I20260620 14:48:04.111392 10966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff85d28489a6462aa681ff3f00d567de" candidate_uuid: "98876ae946d64ece9558d05acc2d5aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21bfa42268ed45ab85db0749ee7f9bd6"
I20260620 14:48:04.111433 11105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff85d28489a6462aa681ff3f00d567de" candidate_uuid: "98876ae946d64ece9558d05acc2d5aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ac13f44d1643d5ac1ff1307f05b869"
I20260620 14:48:04.111526 10966 raft_consensus.cc:3037] T ff85d28489a6462aa681ff3f00d567de P 21bfa42268ed45ab85db0749ee7f9bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:04.111541 11105 raft_consensus.cc:3037] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:04.112637 11105 raft_consensus.cc:2454] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98876ae946d64ece9558d05acc2d5aad in term 1.
I20260620 14:48:04.112637 10966 raft_consensus.cc:2454] T ff85d28489a6462aa681ff3f00d567de P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98876ae946d64ece9558d05acc2d5aad in term 1.
I20260620 14:48:04.112942 11178 leader_election.cc:302] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21bfa42268ed45ab85db0749ee7f9bd6, 98876ae946d64ece9558d05acc2d5aad; no voters:
I20260620 14:48:04.113337 11425 raft_consensus.cc:2781] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:04.113466 11425 raft_consensus.cc:686] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [term 1 LEADER]: Becoming Leader. State: Replica: 98876ae946d64ece9558d05acc2d5aad, State: Running, Role: LEADER
I20260620 14:48:04.113612 11425 consensus_queue.cc:237] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } }
I20260620 14:48:04.115309 10807 catalog_manager.cc:5495] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad reported cstate change: term changed from 0 to 1, leader changed from <none> to 98876ae946d64ece9558d05acc2d5aad (127.8.221.195). New cstate: current_term: 1 leader_uuid: "98876ae946d64ece9558d05acc2d5aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98876ae946d64ece9558d05acc2d5aad" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44945 } health_report { overall_health: HEALTHY } } }
I20260620 14:48:04.134258 11105 raft_consensus.cc:1261] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869 [term 1 FOLLOWER]: Refusing update from remote peer 98876ae946d64ece9558d05acc2d5aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:04.134292 10965 raft_consensus.cc:1261] T ff85d28489a6462aa681ff3f00d567de P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 FOLLOWER]: Refusing update from remote peer 98876ae946d64ece9558d05acc2d5aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:04.134686 11425 consensus_queue.cc:1035] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "21bfa42268ed45ab85db0749ee7f9bd6" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 38463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:04.134826 11410 consensus_queue.cc:1035] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1ac13f44d1643d5ac1ff1307f05b869" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:48:04.152546 11376 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:48:05.720520 10808 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:05.720726 10808 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:05.723599 10808 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9b062d0db4646a18e4eb1b55f8fe5a8: Sending DeleteTablet for 3 replicas of tablet ff85d28489a6462aa681ff3f00d567de
I20260620 14:48:05.724220 11086 tablet_service.cc:1514] Processing DeleteTablet for tablet ff85d28489a6462aa681ff3f00d567de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:05 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:41172
I20260620 14:48:05.724220 10946 tablet_service.cc:1514] Processing DeleteTablet for tablet ff85d28489a6462aa681ff3f00d567de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:05 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:47558
I20260620 14:48:05.724303 11226 tablet_service.cc:1514] Processing DeleteTablet for tablet ff85d28489a6462aa681ff3f00d567de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:05 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:34252
I20260620 14:48:05.724629 11458 tablet_replica.cc:330] T ff85d28489a6462aa681ff3f00d567de P 21bfa42268ed45ab85db0749ee7f9bd6: stopping tablet replica
I20260620 14:48:05.724718 11460 tablet_replica.cc:330] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad: stopping tablet replica
I20260620 14:48:05.724787 11458 raft_consensus.cc:2229] T ff85d28489a6462aa681ff3f00d567de P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:05.724922 11458 raft_consensus.cc:2258] T ff85d28489a6462aa681ff3f00d567de P 21bfa42268ed45ab85db0749ee7f9bd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:05.725145 11459 tablet_replica.cc:330] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869: stopping tablet replica
I20260620 14:48:05.725139 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 10881
I20260620 14:48:05.724848 11460 raft_consensus.cc:2229] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:05.726373 11460 raft_consensus.cc:2258] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:05.727233 11460 ts_tablet_manager.cc:1905] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:05.727566 11459 raft_consensus.cc:2229] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:05.727754 11459 raft_consensus.cc:2258] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:05.728387 11459 ts_tablet_manager.cc:1905] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:05.730064 11460 ts_tablet_manager.cc:1918] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:48:05.730152 11460 log.cc:1196] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/ff85d28489a6462aa681ff3f00d567de
I20260620 14:48:05.730508 11460 ts_tablet_manager.cc:1939] T ff85d28489a6462aa681ff3f00d567de P 98876ae946d64ece9558d05acc2d5aad: Deleting consensus metadata
I20260620 14:48:05.731212 11459 ts_tablet_manager.cc:1918] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:48:05.731237 10793 catalog_manager.cc:4841] TS 98876ae946d64ece9558d05acc2d5aad (127.8.221.195:44945): tablet ff85d28489a6462aa681ff3f00d567de (table test-table [id=56c330dc47324014afbd7a6dc8f6acf5]) successfully deleted
I20260620 14:48:05.731320 11459 log.cc:1196] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/ff85d28489a6462aa681ff3f00d567de
I20260620 14:48:05.731577 11459 ts_tablet_manager.cc:1939] T ff85d28489a6462aa681ff3f00d567de P b1ac13f44d1643d5ac1ff1307f05b869: Deleting consensus metadata
I20260620 14:48:05.732234 10796 catalog_manager.cc:4841] TS b1ac13f44d1643d5ac1ff1307f05b869 (127.8.221.194:42587): tablet ff85d28489a6462aa681ff3f00d567de (table test-table [id=56c330dc47324014afbd7a6dc8f6acf5]) successfully deleted
W20260620 14:48:05.735275 10794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.221.193:38463 (error 108)
I20260620 14:48:05.736357 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 11021
I20260620 14:48:05.740583 11462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.736239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:38463 (local address 127.0.0.1:47582)
0620 14:48:05.740354 (+ 4115us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:05.740458 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.193:38463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":113,"threads_started":1}
W20260620 14:48:05.740756 10794 catalog_manager.cc:4584] TS 21bfa42268ed45ab85db0749ee7f9bd6 (127.8.221.193:38463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ff85d28489a6462aa681ff3f00d567de: Network error: Client connection negotiation failed: client connection to 127.8.221.193:38463: connect: Connection refused (error 111)
I20260620 14:48:05.745901 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 11161
I20260620 14:48:05.754465 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 10777
2026-06-20T14:48:05Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6264 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:05 dist-test-slave-65tz krb5kdc[11465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:05 dist-test-slave-65tz krb5kdc[11465](info): set up 2 sockets
Jun 20 14:48:05 dist-test-slave-65tz krb5kdc[11465](info): commencing operation
krb5kdc: starting...
W20260620 14:48:06.823683 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:48:06 dist-test-slave-65tz krb5kdc[11465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:48:06.957206 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:35077
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:35077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:07.089171 11478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:07.089537 11478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:07.089586 11478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:07.093866 11478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:07.093938 11478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:07.093966 11478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:07.093984 11478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:07.094002 11478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:07.099879 11478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:35077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:35077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:07.101217 11478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:07.102442 11478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:07.112677 11486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:07.112700 11483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:07.112700 11484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:07.112780 11478 server_base.cc:1034] running on GCE node
I20260620 14:48:07.113736 11478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:07.115036 11478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:07.116217 11478 hybrid_clock.cc:648] HybridClock initialized: now 1781966887116185 us; error 46 us; skew 500 ppm
Jun 20 14:48:07 dist-test-slave-65tz krb5kdc[11465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966887, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:07.119450 11478 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:48:07.120445 11478 webserver.cc:460] Webserver started at http://127.8.221.254:39833/ using document root <none> and password file <none>
I20260620 14:48:07.121070 11478 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:07.121140 11478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:07.121393 11478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:07.123721 11478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "69424e07a8024fc8b81cf65e7fdb71e4"
format_stamp: "Formatted at 2026-06-20 14:48:07 on dist-test-slave-65tz"
server_key: "40ce1796207e0bd8ce2814dee9f74d5c"
server_key_iv: "21dd78775c2616f40a0312279c3d6dd7"
server_key_version: "encryptionkey@0"
I20260620 14:48:07.124333 11478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "69424e07a8024fc8b81cf65e7fdb71e4"
format_stamp: "Formatted at 2026-06-20 14:48:07 on dist-test-slave-65tz"
server_key: "40ce1796207e0bd8ce2814dee9f74d5c"
server_key_iv: "21dd78775c2616f40a0312279c3d6dd7"
server_key_version: "encryptionkey@0"
I20260620 14:48:07.128439 11478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:48:07.131222 11493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:07.132879 11478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:48:07.133062 11478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "69424e07a8024fc8b81cf65e7fdb71e4"
format_stamp: "Formatted at 2026-06-20 14:48:07 on dist-test-slave-65tz"
server_key: "40ce1796207e0bd8ce2814dee9f74d5c"
server_key_iv: "21dd78775c2616f40a0312279c3d6dd7"
server_key_version: "encryptionkey@0"
I20260620 14:48:07.133190 11478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:07.177724 11478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:07.181154 11478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:07.181370 11478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:07.189582 11478 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:35077
I20260620 14:48:07.189636 11545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:35077 every 8 connection(s)
I20260620 14:48:07.190732 11478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:07.194540 11546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:07.196821 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 11478
I20260620 14:48:07.197002 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:48:07.197353 9079 external_mini_cluster.cc:1428] Setting key 6ae43dbc0a5421f2e4023ef4c3dd6776
I20260620 14:48:07.202881 11546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4: Bootstrap starting.
I20260620 14:48:07.205705 11546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:07.206609 11546 log.cc:824] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:48:07 dist-test-slave-65tz krb5kdc[11465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:07.208994 11546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4: No bootstrap required, opened a new log
I20260620 14:48:07.213106 11546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69424e07a8024fc8b81cf65e7fdb71e4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 35077 } }
I20260620 14:48:07.213295 11549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.199326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.8.221.254:35077)
0620 14:48:07.199957 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:07.199976 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:07.200019 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:48:07.200884 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:07.200905 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:07.201293 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:07.201690 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:07.202548 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.203635 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.204549 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.205043 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.205229 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.208151 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:07.208157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:07.208173 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:07.208207 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:07.210164 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.210651 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.210658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.210671 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.210795 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.211097 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.211102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.211104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.211547 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:07.211760 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:07.212098 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 14:48:07.212414 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":208,"threads_started":1}
I20260620 14:48:07.213358 11546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:07.213397 11546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69424e07a8024fc8b81cf65e7fdb71e4, State: Initialized, Role: FOLLOWER
I20260620 14:48:07.213976 11546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69424e07a8024fc8b81cf65e7fdb71e4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 35077 } }
I20260620 14:48:07.214118 11546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:07.214241 11546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:07.214378 11546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:07.215699 11546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69424e07a8024fc8b81cf65e7fdb71e4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 35077 } }
I20260620 14:48:07.216135 11546 leader_election.cc:302] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69424e07a8024fc8b81cf65e7fdb71e4; no voters:
I20260620 14:48:07.216513 11546 leader_election.cc:288] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:07.216684 11551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:07.216986 11551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4 [term 1 LEADER]: Becoming Leader. State: Replica: 69424e07a8024fc8b81cf65e7fdb71e4, State: Running, Role: LEADER
I20260620 14:48:07.217355 11551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69424e07a8024fc8b81cf65e7fdb71e4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 35077 } }
I20260620 14:48:07.217979 11546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:07.219401 11553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69424e07a8024fc8b81cf65e7fdb71e4. Latest consensus state: current_term: 1 leader_uuid: "69424e07a8024fc8b81cf65e7fdb71e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69424e07a8024fc8b81cf65e7fdb71e4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 35077 } } }
I20260620 14:48:07.219631 11553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:07.219401 11552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69424e07a8024fc8b81cf65e7fdb71e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69424e07a8024fc8b81cf65e7fdb71e4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 35077 } } }
I20260620 14:48:07.219767 11552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:07.220871 11562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:07.224186 11562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:07.231060 11562 catalog_manager.cc:1347] Generated new cluster ID: c946d5f6b1ae40e686eeb0b1a4a1c7a7
I20260620 14:48:07.231144 11562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:07.250015 11562 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:07.251634 11562 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:07.259441 11562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69424e07a8024fc8b81cf65e7fdb71e4: Generated new TSK 0
I20260620 14:48:07.260452 11562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
I20260620 14:48:07.293200 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:35077
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:07.417623 11572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:07.417994 11572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:07.418081 11572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:07.422406 11572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:07.422477 11572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:07.422537 11572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:48:07.427825 11572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:35077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:07.428874 11572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:07.429901 11572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:07.440037 11580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:07.440135 11578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:07.440038 11577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:07.440482 11572 server_base.cc:1034] running on GCE node
I20260620 14:48:07.441025 11572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:07.441927 11572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:07.443123 11572 hybrid_clock.cc:648] HybridClock initialized: now 1781966887443104 us; error 50 us; skew 500 ppm
Jun 20 14:48:07 dist-test-slave-65tz krb5kdc[11465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966887, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:07.446329 11572 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:48:07.447216 11572 webserver.cc:460] Webserver started at http://127.8.221.193:46401/ using document root <none> and password file <none>
I20260620 14:48:07.447747 11572 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:07.447815 11572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:07.447980 11572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:07.450009 11572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "14c9a55a0e9740639ed649d156b4da9c"
format_stamp: "Formatted at 2026-06-20 14:48:07 on dist-test-slave-65tz"
server_key: "6d2a9b9946b9776b47916bb82bd93b0d"
server_key_iv: "f954a436b904d4516cf0584b02ecd833"
server_key_version: "encryptionkey@0"
I20260620 14:48:07.450523 11572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "14c9a55a0e9740639ed649d156b4da9c"
format_stamp: "Formatted at 2026-06-20 14:48:07 on dist-test-slave-65tz"
server_key: "6d2a9b9946b9776b47916bb82bd93b0d"
server_key_iv: "f954a436b904d4516cf0584b02ecd833"
server_key_version: "encryptionkey@0"
I20260620 14:48:07.454447 11572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:07.457188 11587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:07.458773 11572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:07.458891 11572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "14c9a55a0e9740639ed649d156b4da9c"
format_stamp: "Formatted at 2026-06-20 14:48:07 on dist-test-slave-65tz"
server_key: "6d2a9b9946b9776b47916bb82bd93b0d"
server_key_iv: "f954a436b904d4516cf0584b02ecd833"
server_key_version: "encryptionkey@0"
I20260620 14:48:07.458986 11572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:07.479786 11572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:07.483410 11572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:07.483588 11572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:07.484174 11572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:07.485183 11572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:07.485249 11572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:07.485296 11572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:07.485322 11572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:07.495997 11572 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:33207
I20260620 14:48:07.496101 11700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:33207 every 8 connection(s)
I20260620 14:48:07.496951 11572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:07.501617 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 11572
I20260620 14:48:07.501797 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:48:07.502221 9079 external_mini_cluster.cc:1428] Setting key 4700b1b36c935d416dbb419201f31127
Jun 20 14:48:07 dist-test-slave-65tz krb5kdc[11465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966887, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.194@KRBTEST.COM; defaulting to no policy
I20260620 14:48:07.514181 11549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.499280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:39417 (local address 127.8.221.254:35077)
0620 14:48:07.499417 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:07.499420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:07.500675 (+ 1255us) server_negotiation.cc:407] Connection header received
0620 14:48:07.501660 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:07.501663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:07.501721 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:07.501856 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:07.503721 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.504658 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.506108 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.506377 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.506499 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.510157 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:07.510160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:07.510162 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:07.510187 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:07.511854 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.512438 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.512442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.512443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.512518 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.512872 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.512877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.512879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.513101 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:07.513214 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:07.513839 (+ 625us) server_negotiation.cc:299] Negotiation successful
0620 14:48:07.513980 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:48:07.515108 11703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.499714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:35077 (local address 127.8.221.193:39417)
0620 14:48:07.500500 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:07.500546 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:07.501439 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:07.502179 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:07.502191 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:07.502813 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:07.503501 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.503521 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.504849 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.504854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.505957 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.505970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.506546 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.506550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.507096 (+ 546us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.507927 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:07.507946 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:07.509948 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:07.511992 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:07.511999 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:07.512017 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:07.512307 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:07.512621 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:07.512625 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:07.512626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:07.512774 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:07.513226 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:07.513235 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:07.513562 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 14:48:07.513769 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 14:48:07.514153 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":162,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.194@KRBTEST.COM" created.
I20260620 14:48:07.516741 11701 heartbeater.cc:347] Connected to a master server at 127.8.221.254:35077
I20260620 14:48:07.517127 11701 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:07.517969 11701 heartbeater.cc:511] Master 127.8.221.254:35077 requested a full tablet report, sending...
I20260620 14:48:07.520224 11510 ts_manager.cc:194] Registered new tserver with Master: 14c9a55a0e9740639ed649d156b4da9c (127.8.221.193:33207)
I20260620 14:48:07.521519 11510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:39417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
I20260620 14:48:07.537273 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:35077
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:07.662169 11706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:07.662531 11706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:07.662614 11706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:07.666999 11706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:07.667093 11706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:07.667191 11706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:48:07.673228 11706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:35077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:07.674803 11706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:07.676126 11706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:07.688057 11711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:07.688179 11714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:07.688050 11712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:07.688156 11706 server_base.cc:1034] running on GCE node
I20260620 14:48:07.688822 11706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:07.689587 11706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:07.690812 11706 hybrid_clock.cc:648] HybridClock initialized: now 1781966887690771 us; error 48 us; skew 500 ppm
Jun 20 14:48:07 dist-test-slave-65tz krb5kdc[11465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966887, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:07.693924 11706 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:48:07.694928 11706 webserver.cc:460] Webserver started at http://127.8.221.194:36111/ using document root <none> and password file <none>
I20260620 14:48:07.695519 11706 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:07.695586 11706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:07.695776 11706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:07.698299 11706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "8513e50f9c744cb1bdbdbc9cfbd1814c"
format_stamp: "Formatted at 2026-06-20 14:48:07 on dist-test-slave-65tz"
server_key: "5b5137c5ff8eccd4f74fcdfe39a37ec5"
server_key_iv: "4f7226ffc046c6ca33c9fc5d5205976a"
server_key_version: "encryptionkey@0"
I20260620 14:48:07.698908 11706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "8513e50f9c744cb1bdbdbc9cfbd1814c"
format_stamp: "Formatted at 2026-06-20 14:48:07 on dist-test-slave-65tz"
server_key: "5b5137c5ff8eccd4f74fcdfe39a37ec5"
server_key_iv: "4f7226ffc046c6ca33c9fc5d5205976a"
server_key_version: "encryptionkey@0"
I20260620 14:48:07.703047 11706 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:48:07.705883 11721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:07.707371 11706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:48:07.707489 11706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "8513e50f9c744cb1bdbdbc9cfbd1814c"
format_stamp: "Formatted at 2026-06-20 14:48:07 on dist-test-slave-65tz"
server_key: "5b5137c5ff8eccd4f74fcdfe39a37ec5"
server_key_iv: "4f7226ffc046c6ca33c9fc5d5205976a"
server_key_version: "encryptionkey@0"
I20260620 14:48:07.707660 11706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:07.728463 11706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:07.733012 11706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:07.733222 11706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:07.734040 11706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:07.735165 11706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:07.735229 11706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:07.735278 11706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:07.735294 11706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:07.748163 11706 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:37311
I20260620 14:48:07.748330 11834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:37311 every 8 connection(s)
I20260620 14:48:07.749454 11706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:07.755673 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 11706
I20260620 14:48:07.755812 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:07.756160 9079 external_mini_cluster.cc:1428] Setting key 717b1defd5a4e6fedd65e7d4138954ef
Jun 20 14:48:07 dist-test-slave-65tz krb5kdc[11465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966887, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.195@KRBTEST.COM; defaulting to no policy
I20260620 14:48:07.768603 11549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.751909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:53211 (local address 127.8.221.254:35077)
0620 14:48:07.752125 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:07.752130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:07.753400 (+ 1270us) server_negotiation.cc:407] Connection header received
0620 14:48:07.754737 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:07.754740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:07.754800 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:07.754905 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:07.756866 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.757572 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.759871 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.760099 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.760202 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.763989 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:07.763993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:07.763995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:07.764027 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:07.765583 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.766393 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.766396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.766397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.766484 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.767063 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.767065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.767066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.767273 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:07.767359 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:07.768324 (+ 965us) server_negotiation.cc:299] Negotiation successful
0620 14:48:07.768471 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 14:48:07.770110 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.752299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:35077 (local address 127.8.221.194:53211)
0620 14:48:07.753169 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:07.753223 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:07.754372 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:07.755134 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:07.755142 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:07.755813 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:07.756667 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.756697 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.757736 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.757741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.759701 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.759717 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.760302 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.760310 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.760615 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.761508 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:07.761532 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:07.763751 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:07.765732 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:07.765743 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:07.765768 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:07.766206 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:07.766639 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:07.766643 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:07.766644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:07.766924 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:07.767417 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:07.767433 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:07.767931 (+ 498us) client_negotiation.cc:769] Sending connection context
0620 14:48:07.768218 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 14:48:07.768784 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.195@KRBTEST.COM" created.
I20260620 14:48:07.771961 11835 heartbeater.cc:347] Connected to a master server at 127.8.221.254:35077
I20260620 14:48:07.772426 11835 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:07.773248 11835 heartbeater.cc:511] Master 127.8.221.254:35077 requested a full tablet report, sending...
I20260620 14:48:07.774588 11510 ts_manager.cc:194] Registered new tserver with Master: 8513e50f9c744cb1bdbdbc9cfbd1814c (127.8.221.194:37311)
I20260620 14:48:07.775213 11510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:53211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
I20260620 14:48:07.789561 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:35077
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:39609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:07.937566 11840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:07.938040 11840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:07.938114 11840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:07.942625 11840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:07.942716 11840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:07.942819 11840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:48:07.948716 11840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:39609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:35077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:07.950160 11840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:07.951418 11840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:07.962611 11846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:07.962611 11845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:07.962615 11848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:07.963061 11840 server_base.cc:1034] running on GCE node
I20260620 14:48:07.963641 11840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:07.964392 11840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:07.965623 11840 hybrid_clock.cc:648] HybridClock initialized: now 1781966887965570 us; error 58 us; skew 500 ppm
Jun 20 14:48:07 dist-test-slave-65tz krb5kdc[11465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966887, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:07.968803 11840 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:48:07.969861 11840 webserver.cc:460] Webserver started at http://127.8.221.195:42527/ using document root <none> and password file <none>
I20260620 14:48:07.970511 11840 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:07.970618 11840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:07.970873 11840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:07.973112 11840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "34a5f09786fc48a984e09664ffeda2df"
format_stamp: "Formatted at 2026-06-20 14:48:07 on dist-test-slave-65tz"
server_key: "4a885fda041580eb5e94d8142ba6d21e"
server_key_iv: "98694bf9537991d9111ea9b2a8ed699f"
server_key_version: "encryptionkey@0"
I20260620 14:48:07.973748 11840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "34a5f09786fc48a984e09664ffeda2df"
format_stamp: "Formatted at 2026-06-20 14:48:07 on dist-test-slave-65tz"
server_key: "4a885fda041580eb5e94d8142ba6d21e"
server_key_iv: "98694bf9537991d9111ea9b2a8ed699f"
server_key_version: "encryptionkey@0"
I20260620 14:48:07.978062 11840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:07.980939 11855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:07.982578 11840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:07.982758 11840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "34a5f09786fc48a984e09664ffeda2df"
format_stamp: "Formatted at 2026-06-20 14:48:07 on dist-test-slave-65tz"
server_key: "4a885fda041580eb5e94d8142ba6d21e"
server_key_iv: "98694bf9537991d9111ea9b2a8ed699f"
server_key_version: "encryptionkey@0"
I20260620 14:48:07.982882 11840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:08.028016 11840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:08.031276 11840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:08.031514 11840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:08.032145 11840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:08.033013 11840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:08.033085 11840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:08.033154 11840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:08.033205 11840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:08.044162 11840 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:38231
I20260620 14:48:08.044193 11968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:38231 every 8 connection(s)
I20260620 14:48:08.045113 11840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:08.048583 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 11840
I20260620 14:48:08.048719 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:08.049060 9079 external_mini_cluster.cc:1428] Setting key 60a275f02e3faac174bef23e018cf834
Jun 20 14:48:08 dist-test-slave-65tz krb5kdc[11465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966887, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:08.060303 11549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:08.047379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:39801 (local address 127.8.221.254:35077)
0620 14:48:08.047514 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:08.047518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:08.048826 (+ 1308us) server_negotiation.cc:407] Connection header received
0620 14:48:08.050091 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:08.050095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:08.050149 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:08.050244 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:08.051534 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:08.052166 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:08.053396 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:08.053636 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:08.053728 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:08.056652 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:08.056655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:08.056658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:08.056684 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:08.058264 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:08.058813 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:08.058815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:08.058816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:08.058893 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:08.059226 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:08.059228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:08.059229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:08.059425 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:08.059509 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:08.060028 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 14:48:08.060147 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:48:08.061256 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:08.047793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:35077 (local address 127.8.221.195:39801)
0620 14:48:08.048615 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:08.048687 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:08.049778 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:08.050437 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:08.050444 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:08.050804 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:08.051350 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:08.051367 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:08.052310 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:08.052322 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:08.053266 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:08.053275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:08.053848 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:08.053856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:08.054016 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:08.054703 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:08.054739 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:08.056459 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:08.058413 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:08.058418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:08.058431 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:08.058710 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:08.058998 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:08.059001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:08.059002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:08.059134 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:08.059524 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:08.059529 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:08.059790 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 14:48:08.059962 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 14:48:08.060240 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":172,"threads_started":1}
I20260620 14:48:08.063103 11969 heartbeater.cc:347] Connected to a master server at 127.8.221.254:35077
I20260620 14:48:08.063512 11969 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:08.064185 11969 heartbeater.cc:511] Master 127.8.221.254:35077 requested a full tablet report, sending...
I20260620 14:48:08.065387 11510 ts_manager.cc:194] Registered new tserver with Master: 34a5f09786fc48a984e09664ffeda2df (127.8.221.195:38231)
I20260620 14:48:08.066063 11510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:39801
I20260620 14:48:08.076483 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:48:08.094298 11549 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:08.092539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.8.221.254:35077)
0620 14:48:08.092708 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:08.092712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:08.092911 (+ 199us) server_negotiation.cc:407] Connection header received
0620 14:48:08.093065 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:08.093069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:08.093126 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:08.093237 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:08.094144 (+ 907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44970: BlockingRecv error: recv got EOF from 127.0.0.1:44970 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:48:08.095587 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 11572
I20260620 14:48:08.102682 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 11706
I20260620 14:48:08.108785 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 11840
I20260620 14:48:08.115288 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 11478
2026-06-20T14:48:08Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2359 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:08 dist-test-slave-65tz krb5kdc[11981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:08 dist-test-slave-65tz krb5kdc[11981](info): set up 2 sockets
Jun 20 14:48:08 dist-test-slave-65tz krb5kdc[11981](info): commencing operation
krb5kdc: starting...
W20260620 14:48:09.160172 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:48:09 dist-test-slave-65tz krb5kdc[11981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:48:09.298607 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:34549
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:32961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:34549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:09.419806 11994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:09.420094 11994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:09.420137 11994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:09.424219 11994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:09.424293 11994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:09.424314 11994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:09.424333 11994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:09.424350 11994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:09.429634 11994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:32961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:34549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:34549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:09.430774 11994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:09.431771 11994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:09.440687 11999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:09.440687 12000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:09.440737 12002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:09.441124 11994 server_base.cc:1034] running on GCE node
I20260620 14:48:09.441999 11994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:09.443408 11994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:09.444593 11994 hybrid_clock.cc:648] HybridClock initialized: now 1781966889444556 us; error 59 us; skew 500 ppm
Jun 20 14:48:09 dist-test-slave-65tz krb5kdc[11981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966889, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:09.452625 11994 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:48:09.453917 11994 webserver.cc:460] Webserver started at http://127.8.221.254:42741/ using document root <none> and password file <none>
I20260620 14:48:09.454623 11994 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:09.454722 11994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:09.454996 11994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:09.457708 11994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "d863e3cc6d8e40d7b79fadff062938c1"
format_stamp: "Formatted at 2026-06-20 14:48:09 on dist-test-slave-65tz"
server_key: "e634e5db6d77ce8cdf6bd7e5d8f77f89"
server_key_iv: "d09376404018ccc96a746e830bf73a01"
server_key_version: "encryptionkey@0"
I20260620 14:48:09.458518 11994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "d863e3cc6d8e40d7b79fadff062938c1"
format_stamp: "Formatted at 2026-06-20 14:48:09 on dist-test-slave-65tz"
server_key: "e634e5db6d77ce8cdf6bd7e5d8f77f89"
server_key_iv: "d09376404018ccc96a746e830bf73a01"
server_key_version: "encryptionkey@0"
I20260620 14:48:09.463030 11994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:48:09.466099 12009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:09.467789 11994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:48:09.467978 11994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "d863e3cc6d8e40d7b79fadff062938c1"
format_stamp: "Formatted at 2026-06-20 14:48:09 on dist-test-slave-65tz"
server_key: "e634e5db6d77ce8cdf6bd7e5d8f77f89"
server_key_iv: "d09376404018ccc96a746e830bf73a01"
server_key_version: "encryptionkey@0"
I20260620 14:48:09.468120 11994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:09.497103 11994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:09.500944 11994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:09.501217 11994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:09.510885 11994 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:34549
I20260620 14:48:09.511250 12061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:34549 every 8 connection(s)
I20260620 14:48:09.512215 11994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:09.515498 12062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:09.517709 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 11994
I20260620 14:48:09.517870 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:48:09.518214 9079 external_mini_cluster.cc:1428] Setting key cc1ecff1475de4a6f541fdcff2dd55a3
I20260620 14:48:09.523557 12062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1: Bootstrap starting.
I20260620 14:48:09.526443 12062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:09.527421 12062 log.cc:824] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:48:09 dist-test-slave-65tz krb5kdc[11981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:09.529759 12062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1: No bootstrap required, opened a new log
I20260620 14:48:09.534039 12062 raft_consensus.cc:348] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d863e3cc6d8e40d7b79fadff062938c1" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 34549 } }
I20260620 14:48:09.534375 12062 raft_consensus.cc:374] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:09.534456 12062 raft_consensus.cc:729] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d863e3cc6d8e40d7b79fadff062938c1, State: Initialized, Role: FOLLOWER
I20260620 14:48:09.534969 12062 consensus_queue.cc:260] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d863e3cc6d8e40d7b79fadff062938c1" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 34549 } }
I20260620 14:48:09.535161 12062 raft_consensus.cc:388] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:09.535267 12062 raft_consensus.cc:482] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:09.535419 12062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:09.536041 12065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.519907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.8.221.254:34549)
0620 14:48:09.520405 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:09.520429 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:09.520475 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:48:09.521340 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:09.521361 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:09.521775 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:09.522208 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:09.523265 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.524545 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.525959 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.526720 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.526891 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:09.530239 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:09.530246 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:09.530264 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:09.530335 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:09.532409 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:09.532924 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:09.532931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:09.532936 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:09.533063 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:09.533363 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:09.533373 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:09.533374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:09.533946 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:09.534196 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:09.534490 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 14:48:09.534888 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":143,"threads_started":1}
I20260620 14:48:09.536557 12062 raft_consensus.cc:504] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d863e3cc6d8e40d7b79fadff062938c1" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 34549 } }
I20260620 14:48:09.536969 12062 leader_election.cc:302] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d863e3cc6d8e40d7b79fadff062938c1; no voters:
I20260620 14:48:09.537379 12062 leader_election.cc:288] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:09.537539 12067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:09.537855 12067 raft_consensus.cc:686] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1 [term 1 LEADER]: Becoming Leader. State: Replica: d863e3cc6d8e40d7b79fadff062938c1, State: Running, Role: LEADER
I20260620 14:48:09.538255 12067 consensus_queue.cc:237] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d863e3cc6d8e40d7b79fadff062938c1" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 34549 } }
I20260620 14:48:09.538897 12062 sys_catalog.cc:564] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:09.539785 12067 sys_catalog.cc:455] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d863e3cc6d8e40d7b79fadff062938c1. Latest consensus state: current_term: 1 leader_uuid: "d863e3cc6d8e40d7b79fadff062938c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d863e3cc6d8e40d7b79fadff062938c1" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 34549 } } }
I20260620 14:48:09.539891 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:48:09.539944 12067 sys_catalog.cc:458] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:09.540577 12069 sys_catalog.cc:455] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d863e3cc6d8e40d7b79fadff062938c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d863e3cc6d8e40d7b79fadff062938c1" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 34549 } } }
I20260620 14:48:09.540688 12069 sys_catalog.cc:458] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1 [sys.catalog]: This master's current role is: LEADER
W20260620 14:48:09.544626 12082 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:48:09.544713 12082 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:48:09.544787 12076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:09.545637 12076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:09.553836 12076 catalog_manager.cc:1347] Generated new cluster ID: 91eaec20bfaf427b8876ab740f7d605f
I20260620 14:48:09.553916 12076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:09.560974 12065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.554440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.8.221.254:34549)
0620 14:48:09.554578 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:09.554584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:09.554936 (+ 352us) server_negotiation.cc:407] Connection header received
0620 14:48:09.555097 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:09.555100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:09.555169 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:09.555271 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:09.555923 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.556834 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.558171 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.558584 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.558697 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:09.559742 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:09.559746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:09.559750 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:09.560430 (+ 680us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:09.560469 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:09.560490 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:09.560590 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:09.560718 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 14:48:09.560799 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:48:09.579555 12076 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:09.580646 12076 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:09.586589 12076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d863e3cc6d8e40d7b79fadff062938c1: Generated new TSK 0
I20260620 14:48:09.587339 12076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:48:09.597739 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 11994
2026-06-20T14:48:09Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1478 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:09 dist-test-slave-65tz krb5kdc[12095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:09 dist-test-slave-65tz krb5kdc[12095](info): set up 2 sockets
Jun 20 14:48:09 dist-test-slave-65tz krb5kdc[12095](info): commencing operation
krb5kdc: starting...
W20260620 14:48:10.643798 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:48:10 dist-test-slave-65tz krb5kdc[12095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:48:10.777431 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:46509
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:46509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:10.910328 12108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:10.910696 12108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:10.910827 12108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:10.916026 12108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:10.916101 12108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:10.916124 12108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:10.916144 12108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:10.916167 12108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:10.922364 12108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:46509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:46509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:10.923676 12108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:10.924916 12108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:10.934963 12114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:10.935009 12113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:10.935005 12116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:10.935269 12108 server_base.cc:1034] running on GCE node
I20260620 14:48:10.935987 12108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:10.937266 12108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:10.938449 12108 hybrid_clock.cc:648] HybridClock initialized: now 1781966890938430 us; error 32 us; skew 500 ppm
Jun 20 14:48:10 dist-test-slave-65tz krb5kdc[12095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966890, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:10.941514 12108 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:48:10.942512 12108 webserver.cc:460] Webserver started at http://127.8.221.254:46583/ using document root <none> and password file <none>
I20260620 14:48:10.943140 12108 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:10.943228 12108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:10.943460 12108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:10.945649 12108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "b023501c7aee4d8b803f25f4a4b417f3"
format_stamp: "Formatted at 2026-06-20 14:48:10 on dist-test-slave-65tz"
server_key: "ba74e9f0af4afedb138b679480f44902"
server_key_iv: "d31a4139e3e9ad41e838268a8f7ac89e"
server_key_version: "encryptionkey@0"
I20260620 14:48:10.947717 12108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "b023501c7aee4d8b803f25f4a4b417f3"
format_stamp: "Formatted at 2026-06-20 14:48:10 on dist-test-slave-65tz"
server_key: "ba74e9f0af4afedb138b679480f44902"
server_key_iv: "d31a4139e3e9ad41e838268a8f7ac89e"
server_key_version: "encryptionkey@0"
I20260620 14:48:10.952191 12108 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:10.955164 12123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:10.956864 12108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:10.957018 12108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "b023501c7aee4d8b803f25f4a4b417f3"
format_stamp: "Formatted at 2026-06-20 14:48:10 on dist-test-slave-65tz"
server_key: "ba74e9f0af4afedb138b679480f44902"
server_key_iv: "d31a4139e3e9ad41e838268a8f7ac89e"
server_key_version: "encryptionkey@0"
I20260620 14:48:10.957144 12108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:10.988109 12108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:10.992022 12108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:10.992233 12108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:11.000093 12108 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:46509
I20260620 14:48:11.000397 12175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:46509 every 8 connection(s)
I20260620 14:48:11.001210 12108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:11.004834 12176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:11.007890 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 12108
I20260620 14:48:11.008011 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:48:11.008318 9079 external_mini_cluster.cc:1428] Setting key 905ec3da8560d4f139a14dbeaade6328
I20260620 14:48:11.014091 12176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3: Bootstrap starting.
I20260620 14:48:11.017405 12176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:11.018420 12176 log.cc:824] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:48:11 dist-test-slave-65tz krb5kdc[12095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:11.021260 12176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3: No bootstrap required, opened a new log
I20260620 14:48:11.026104 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:11.010097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.8.221.254:46509)
0620 14:48:11.010646 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:11.010669 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:11.010732 (+ 63us) server_negotiation.cc:407] Connection header received
0620 14:48:11.011786 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:11.011816 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:11.012350 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:11.012838 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:11.013617 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:11.015218 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:11.016194 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:11.016914 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:11.017091 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:11.020031 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:11.020038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:11.020058 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:11.020100 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:11.022329 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:11.022813 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:11.022819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:11.022825 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:11.022976 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:11.023301 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:11.023304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:11.023305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:11.023863 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:11.024084 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:11.024502 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 14:48:11.024914 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":140,"threads_started":1}
I20260620 14:48:11.027565 12176 raft_consensus.cc:348] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b023501c7aee4d8b803f25f4a4b417f3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 46509 } }
I20260620 14:48:11.028012 12176 raft_consensus.cc:374] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:11.028090 12176 raft_consensus.cc:729] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b023501c7aee4d8b803f25f4a4b417f3, State: Initialized, Role: FOLLOWER
I20260620 14:48:11.028635 12176 consensus_queue.cc:260] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b023501c7aee4d8b803f25f4a4b417f3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 46509 } }
I20260620 14:48:11.028822 12176 raft_consensus.cc:388] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:11.028895 12176 raft_consensus.cc:482] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:11.029016 12176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:11.128682 12176 raft_consensus.cc:504] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b023501c7aee4d8b803f25f4a4b417f3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 46509 } }
I20260620 14:48:11.129377 12176 leader_election.cc:302] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b023501c7aee4d8b803f25f4a4b417f3; no voters:
I20260620 14:48:11.129941 12176 leader_election.cc:288] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:11.130158 12181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:11.130522 12181 raft_consensus.cc:686] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3 [term 1 LEADER]: Becoming Leader. State: Replica: b023501c7aee4d8b803f25f4a4b417f3, State: Running, Role: LEADER
I20260620 14:48:11.131711 12176 sys_catalog.cc:564] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:11.131677 12181 consensus_queue.cc:237] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b023501c7aee4d8b803f25f4a4b417f3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 46509 } }
I20260620 14:48:11.133666 12183 sys_catalog.cc:455] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b023501c7aee4d8b803f25f4a4b417f3. Latest consensus state: current_term: 1 leader_uuid: "b023501c7aee4d8b803f25f4a4b417f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b023501c7aee4d8b803f25f4a4b417f3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 46509 } } }
I20260620 14:48:11.133845 12183 sys_catalog.cc:458] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:11.134235 12182 sys_catalog.cc:455] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b023501c7aee4d8b803f25f4a4b417f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b023501c7aee4d8b803f25f4a4b417f3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 46509 } } }
I20260620 14:48:11.134678 12182 sys_catalog.cc:458] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3 [sys.catalog]: This master's current role is: LEADER
W20260620 14:48:11.137506 12196 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:48:11.137603 12196 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:48:11.137692 12195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:11.138581 12195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:11.147665 12195 catalog_manager.cc:1347] Generated new cluster ID: 95c336c096944c4895af7e9a85c00fd5
I20260620 14:48:11.147774 12195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:11.169040 12195 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:11.170596 12195 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:11.184324 12195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b023501c7aee4d8b803f25f4a4b417f3: Generated new TSK 0
I20260620 14:48:11.185091 12195 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
W20260620 14:48:12.380348 9079 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.8.221.193: real 1.179s user 0.000s sys 0.004s
I20260620 14:48:12.381089 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:46509
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:12.499117 12202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:12.499440 12202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:12.499514 12202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:12.503814 12202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:12.503931 12202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:12.504032 12202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:48:12.509672 12202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:12.511216 12202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:12.512322 12202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:12.527477 12207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:12.527469 12210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:12.527503 12208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:12.527930 12202 server_base.cc:1034] running on GCE node
I20260620 14:48:12.528527 12202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:12.529220 12202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:12.530398 12202 hybrid_clock.cc:648] HybridClock initialized: now 1781966892530373 us; error 60 us; skew 500 ppm
Jun 20 14:48:12 dist-test-slave-65tz krb5kdc[12095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966892, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:12.533764 12202 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:48:12.535000 12202 webserver.cc:460] Webserver started at http://127.8.221.193:38767/ using document root <none> and password file <none>
I20260620 14:48:12.535719 12202 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:12.535792 12202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:12.536037 12202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:12.538669 12202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "75b701279f594e80b27c57611ab7b8db"
format_stamp: "Formatted at 2026-06-20 14:48:12 on dist-test-slave-65tz"
server_key: "a685e90c4108b87b53d310fed2b6a441"
server_key_iv: "08ff3f83887234355f8e765cd8eb9330"
server_key_version: "encryptionkey@0"
I20260620 14:48:12.539321 12202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "75b701279f594e80b27c57611ab7b8db"
format_stamp: "Formatted at 2026-06-20 14:48:12 on dist-test-slave-65tz"
server_key: "a685e90c4108b87b53d310fed2b6a441"
server_key_iv: "08ff3f83887234355f8e765cd8eb9330"
server_key_version: "encryptionkey@0"
I20260620 14:48:12.544104 12202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:48:12.547181 12217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:12.548787 12202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 14:48:12.548918 12202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "75b701279f594e80b27c57611ab7b8db"
format_stamp: "Formatted at 2026-06-20 14:48:12 on dist-test-slave-65tz"
server_key: "a685e90c4108b87b53d310fed2b6a441"
server_key_iv: "08ff3f83887234355f8e765cd8eb9330"
server_key_version: "encryptionkey@0"
I20260620 14:48:12.549038 12202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:12.582485 12202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:12.586508 12202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:12.586760 12202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:12.587656 12202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:12.588663 12202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:12.588716 12202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:12.588763 12202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:12.588779 12202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:12.601256 12202 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:34581
I20260620 14:48:12.601372 12330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:34581 every 8 connection(s)
I20260620 14:48:12.602571 12202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:12.608904 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 12202
I20260620 14:48:12.609042 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:48:12.609351 9079 external_mini_cluster.cc:1428] Setting key 8cafc3266b22925179f93ad4f89c8e6b
Jun 20 14:48:12 dist-test-slave-65tz krb5kdc[12095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966892, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.194@KRBTEST.COM; defaulting to no policy
I20260620 14:48:12.620355 12333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:12.604891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:44451 (local address 127.8.221.254:46509)
0620 14:48:12.605228 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:12.605234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:12.606278 (+ 1044us) server_negotiation.cc:407] Connection header received
0620 14:48:12.607759 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:12.607762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:12.607836 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:12.607949 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:12.609613 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:12.610359 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:12.611747 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:12.611997 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:12.612124 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:12.615642 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:12.615646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:12.615648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:12.615679 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:12.617532 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:12.618228 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:12.618234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:12.618236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:12.618328 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:12.618824 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:12.618831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:12.618834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:12.619210 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:12.619333 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:12.620015 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 14:48:12.620163 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":75,"threads_started":1}
I20260620 14:48:12.621488 12334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:12.605413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:46509 (local address 127.8.221.193:44451)
0620 14:48:12.606085 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:12.606139 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:12.607521 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:12.608148 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:12.608156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:12.608570 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:12.609354 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:12.609375 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:12.610545 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:12.610548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:12.611601 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:12.611614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:12.612176 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:12.612181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:12.612385 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:12.613233 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:12.613252 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:12.615415 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:12.617673 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:12.617680 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:12.617702 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:12.618114 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:12.618433 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:12.618437 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:12.618439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:12.618611 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:12.619364 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:12.619371 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:12.619726 (+ 355us) client_negotiation.cc:769] Sending connection context
0620 14:48:12.619930 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 14:48:12.620362 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":124,"threads_started":1}
I20260620 14:48:12.623191 12331 heartbeater.cc:347] Connected to a master server at 127.8.221.254:46509
I20260620 14:48:12.623499 12331 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:12.624089 12331 heartbeater.cc:511] Master 127.8.221.254:46509 requested a full tablet report, sending...
I20260620 14:48:12.626096 12139 ts_manager.cc:194] Registered new tserver with Master: 75b701279f594e80b27c57611ab7b8db (127.8.221.193:34581)
I20260620 14:48:12.627925 12139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:44451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.194@KRBTEST.COM" created.
I20260620 14:48:13.630676 12331 heartbeater.cc:503] Master 127.8.221.254:46509 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
W20260620 14:48:13.695766 9079 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.8.221.194: real 1.085s user 0.000s sys 0.004s
I20260620 14:48:13.696373 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:46509
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:13.810196 12337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:13.810491 12337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:13.810536 12337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:13.814771 12337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:13.814841 12337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:13.814954 12337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:48:13.820434 12337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:13.821565 12337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:13.822660 12337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:13.833456 12343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:13.833489 12345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:13.833906 12342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:13.834183 12337 server_base.cc:1034] running on GCE node
I20260620 14:48:13.834754 12337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:13.835635 12337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:13.836791 12337 hybrid_clock.cc:648] HybridClock initialized: now 1781966893836745 us; error 61 us; skew 500 ppm
Jun 20 14:48:13 dist-test-slave-65tz krb5kdc[12095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966893, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:13.840406 12337 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:48:13.841535 12337 webserver.cc:460] Webserver started at http://127.8.221.194:42687/ using document root <none> and password file <none>
I20260620 14:48:13.842295 12337 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:13.842370 12337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:13.842626 12337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:13.845425 12337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "05446b0010a94b7d98d622ebb59b2353"
format_stamp: "Formatted at 2026-06-20 14:48:13 on dist-test-slave-65tz"
server_key: "b88cc8647c60a105e05362b9c2432e34"
server_key_iv: "37a3b6a6e35cff6676782188bcec4ae5"
server_key_version: "encryptionkey@0"
I20260620 14:48:13.846102 12337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "05446b0010a94b7d98d622ebb59b2353"
format_stamp: "Formatted at 2026-06-20 14:48:13 on dist-test-slave-65tz"
server_key: "b88cc8647c60a105e05362b9c2432e34"
server_key_iv: "37a3b6a6e35cff6676782188bcec4ae5"
server_key_version: "encryptionkey@0"
I20260620 14:48:13.850874 12337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:48:13.854046 12352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:13.855768 12337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:48:13.855922 12337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "05446b0010a94b7d98d622ebb59b2353"
format_stamp: "Formatted at 2026-06-20 14:48:13 on dist-test-slave-65tz"
server_key: "b88cc8647c60a105e05362b9c2432e34"
server_key_iv: "37a3b6a6e35cff6676782188bcec4ae5"
server_key_version: "encryptionkey@0"
I20260620 14:48:13.856055 12337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:13.874183 12337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:13.877887 12337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:13.878113 12337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:13.878786 12337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:13.879671 12337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:13.879742 12337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:13.879802 12337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:13.879848 12337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:13.890985 12337 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:35307
I20260620 14:48:13.892258 12337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:13.892593 12465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:35307 every 8 connection(s)
I20260620 14:48:13.893208 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 12337
I20260620 14:48:13.893378 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:13.893749 9079 external_mini_cluster.cc:1428] Setting key 92a6e24e564a8b2fca794893e869041e
Jun 20 14:48:13 dist-test-slave-65tz krb5kdc[12095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966893, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.195@KRBTEST.COM; defaulting to no policy
I20260620 14:48:13.909339 12468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:13.894617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:53773 (local address 127.8.221.254:46509)
0620 14:48:13.894861 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:13.894866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:13.895993 (+ 1127us) server_negotiation.cc:407] Connection header received
0620 14:48:13.897670 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:13.897674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:13.897730 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:13.897873 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:13.899095 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.899981 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.901740 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.902247 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.902392 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:13.905743 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:13.905750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:13.905755 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:13.905832 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:13.907297 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:13.907819 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:13.907822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:13.907823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:13.907923 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:13.908278 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:13.908282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:13.908283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:13.908460 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:13.908535 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:13.909094 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 14:48:13.909199 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":84,"threads_started":1}
I20260620 14:48:13.910562 12469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:13.895186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:46509 (local address 127.8.221.194:53773)
0620 14:48:13.895845 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:13.895893 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:13.897432 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:13.898056 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:13.898063 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:13.898434 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:13.898956 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:13.898969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.900170 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.900175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:13.901558 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:13.901577 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.902436 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.902440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:13.902668 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:13.903767 (+ 1099us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:13.903794 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:13.905491 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:13.907411 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:13.907417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:13.907433 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:13.907719 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:13.908024 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:13.908027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:13.908028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:13.908196 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:13.908559 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:13.908566 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:13.908868 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 14:48:13.909091 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 14:48:13.909422 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":205,"threads_started":1}
I20260620 14:48:13.912106 12466 heartbeater.cc:347] Connected to a master server at 127.8.221.254:46509
I20260620 14:48:13.912375 12466 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:13.912950 12466 heartbeater.cc:511] Master 127.8.221.254:46509 requested a full tablet report, sending...
I20260620 14:48:13.914237 12139 ts_manager.cc:194] Registered new tserver with Master: 05446b0010a94b7d98d622ebb59b2353 (127.8.221.194:35307)
I20260620 14:48:13.914862 12139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:53773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.195@KRBTEST.COM" created.
I20260620 14:48:14.917299 12466 heartbeater.cc:503] Master 127.8.221.254:46509 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
W20260620 14:48:15.016722 9079 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.8.221.195: real 1.121s user 0.002s sys 0.003s
I20260620 14:48:15.017277 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:46509
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:15.143580 12472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:15.143889 12472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:15.143939 12472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:15.148305 12472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:15.148416 12472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:15.148492 12472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:48:15.155380 12472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:15.156738 12472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:15.158080 12472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:15.170071 12478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:15.170064 12477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:15.170174 12472 server_base.cc:1034] running on GCE node
W20260620 14:48:15.170066 12480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:15.170882 12472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:15.171689 12472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:15.172891 12472 hybrid_clock.cc:648] HybridClock initialized: now 1781966895172845 us; error 56 us; skew 500 ppm
Jun 20 14:48:15 dist-test-slave-65tz krb5kdc[12095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966895, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:15.180013 12472 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:48:15.181193 12472 webserver.cc:460] Webserver started at http://127.8.221.195:40895/ using document root <none> and password file <none>
I20260620 14:48:15.181945 12472 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:15.182045 12472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:15.182310 12472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:15.184769 12472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "4348861abf7f41989d6aa4adb042115b"
format_stamp: "Formatted at 2026-06-20 14:48:15 on dist-test-slave-65tz"
server_key: "ccbd1f183e11242e6a1001748a1ba680"
server_key_iv: "fdafc2127a20e835a9f021676e5e7760"
server_key_version: "encryptionkey@0"
I20260620 14:48:15.185423 12472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "4348861abf7f41989d6aa4adb042115b"
format_stamp: "Formatted at 2026-06-20 14:48:15 on dist-test-slave-65tz"
server_key: "ccbd1f183e11242e6a1001748a1ba680"
server_key_iv: "fdafc2127a20e835a9f021676e5e7760"
server_key_version: "encryptionkey@0"
I20260620 14:48:15.190029 12472 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 14:48:15.192914 12487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:15.194600 12472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:15.194753 12472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "4348861abf7f41989d6aa4adb042115b"
format_stamp: "Formatted at 2026-06-20 14:48:15 on dist-test-slave-65tz"
server_key: "ccbd1f183e11242e6a1001748a1ba680"
server_key_iv: "fdafc2127a20e835a9f021676e5e7760"
server_key_version: "encryptionkey@0"
I20260620 14:48:15.194882 12472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:15.207082 12472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:15.210443 12472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:15.210672 12472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:15.211377 12472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:15.212359 12472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:15.212430 12472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:15.212502 12472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:15.212543 12472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:15.223677 12472 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:35367
I20260620 14:48:15.223708 12600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:35367 every 8 connection(s)
I20260620 14:48:15.224830 12472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:15.235183 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 12472
I20260620 14:48:15.235332 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:15.235661 9079 external_mini_cluster.cc:1428] Setting key e6973532143b0e04403a2b5ea0318caa
Jun 20 14:48:15 dist-test-slave-65tz krb5kdc[12095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966895, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:15.240597 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:15.227113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:55441 (local address 127.8.221.254:46509)
0620 14:48:15.227439 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:15.227443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:15.228319 (+ 876us) server_negotiation.cc:407] Connection header received
0620 14:48:15.229746 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:15.229748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:15.229872 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:15.229989 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:15.231252 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.231995 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.233377 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.233617 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.233711 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:15.236534 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:15.236539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:15.236542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:15.236573 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:15.238545 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:15.239069 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:15.239074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:15.239076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:15.239165 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:15.239479 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:15.239484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:15.239486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:15.239688 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:15.239789 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:15.240277 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 14:48:15.240410 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":110,"threads_started":1}
I20260620 14:48:15.241408 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:15.227601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:46509 (local address 127.8.221.195:55441)
0620 14:48:15.228134 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:15.228185 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:15.229512 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:15.230187 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:15.230194 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:15.230576 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:15.231105 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:15.231117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.232109 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.232112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:15.233231 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:15.233240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.233815 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.233822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:15.233962 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:15.234560 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:15.234576 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:15.236302 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:15.238675 (+ 2373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:15.238681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:15.238699 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:15.238965 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:15.239253 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:15.239258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:15.239260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:15.239390 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:15.239785 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:15.239791 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:15.240034 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 14:48:15.240220 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 14:48:15.240540 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":114,"threads_started":1}
I20260620 14:48:15.242588 12601 heartbeater.cc:347] Connected to a master server at 127.8.221.254:46509
I20260620 14:48:15.242813 12601 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:15.243326 12601 heartbeater.cc:511] Master 127.8.221.254:46509 requested a full tablet report, sending...
I20260620 14:48:15.244328 12139 ts_manager.cc:194] Registered new tserver with Master: 4348861abf7f41989d6aa4adb042115b (127.8.221.195:35367)
I20260620 14:48:15.244863 12139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:55441
I20260620 14:48:15.250170 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:48:15 dist-test-slave-65tz krb5kdc[12095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966895, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 14:48:15 dist-test-slave-65tz krb5kdc[12095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966895, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:15.279022 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:15.270476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33874 (local address 127.8.221.254:46509)
0620 14:48:15.270678 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:15.270684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:15.270802 (+ 118us) server_negotiation.cc:407] Connection header received
0620 14:48:15.270995 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:15.270998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:15.271043 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:15.271127 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:15.271861 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.272397 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.273231 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.273464 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.273569 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:15.275996 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:15.276000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:15.276002 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:15.276023 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:15.277320 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:15.277707 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:15.277713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:15.277715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:15.277841 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:15.278110 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:15.278114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:15.278116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:15.278337 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:15.278474 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:15.278722 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 14:48:15.278854 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 14:48:15.279527 12139 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33874
I20260620 14:48:15.280666 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 12202
I20260620 14:48:15.286664 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 12337
I20260620 14:48:15.293210 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 12472
I20260620 14:48:15.298933 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 12108
2026-06-20T14:48:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (5704 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:15 dist-test-slave-65tz krb5kdc[12614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:15 dist-test-slave-65tz krb5kdc[12614](info): set up 2 sockets
Jun 20 14:48:15 dist-test-slave-65tz krb5kdc[12614](info): commencing operation
krb5kdc: starting...
W20260620 14:48:16.351343 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:48:16 dist-test-slave-65tz krb5kdc[12614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:48:16.483127 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:35571
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:35571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:16.597414 12627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:16.597716 12627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:16.597759 12627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:16.601905 12627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:16.601969 12627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:16.601989 12627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:16.602046 12627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:16.602090 12627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:16.607553 12627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:35571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:35571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:16.608676 12627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:16.609712 12627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:16.618590 12633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:16.618697 12635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:16.618849 12632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:16.618880 12627 server_base.cc:1034] running on GCE node
I20260620 14:48:16.619565 12627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:16.620688 12627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:16.621917 12627 hybrid_clock.cc:648] HybridClock initialized: now 1781966896621889 us; error 48 us; skew 500 ppm
Jun 20 14:48:16 dist-test-slave-65tz krb5kdc[12614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966896, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:16.625058 12627 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:48:16.626185 12627 webserver.cc:460] Webserver started at http://127.8.221.254:45773/ using document root <none> and password file <none>
I20260620 14:48:16.626804 12627 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:16.626900 12627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:16.627158 12627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:16.629329 12627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "a1852049c917409ca179acf6f83f9768"
format_stamp: "Formatted at 2026-06-20 14:48:16 on dist-test-slave-65tz"
server_key: "b02d682c5de4182ac2f451c00aedcf5c"
server_key_iv: "70e7cf9d06ebb98a515549716cd42705"
server_key_version: "encryptionkey@0"
I20260620 14:48:16.630010 12627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "a1852049c917409ca179acf6f83f9768"
format_stamp: "Formatted at 2026-06-20 14:48:16 on dist-test-slave-65tz"
server_key: "b02d682c5de4182ac2f451c00aedcf5c"
server_key_iv: "70e7cf9d06ebb98a515549716cd42705"
server_key_version: "encryptionkey@0"
I20260620 14:48:16.634236 12627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:16.636809 12642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:16.638274 12627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:16.638422 12627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "a1852049c917409ca179acf6f83f9768"
format_stamp: "Formatted at 2026-06-20 14:48:16 on dist-test-slave-65tz"
server_key: "b02d682c5de4182ac2f451c00aedcf5c"
server_key_iv: "70e7cf9d06ebb98a515549716cd42705"
server_key_version: "encryptionkey@0"
I20260620 14:48:16.638533 12627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:16.656944 12627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:16.659873 12627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:16.660080 12627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:16.667522 12627 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:35571
I20260620 14:48:16.667577 12694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:35571 every 8 connection(s)
I20260620 14:48:16.668500 12627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:16.669631 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 12627
I20260620 14:48:16.669787 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:48:16.670171 9079 external_mini_cluster.cc:1428] Setting key 9a07420677ce3200e8de7bea20c7e576
I20260620 14:48:16.672578 12695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:16.679472 12695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768: Bootstrap starting.
Jun 20 14:48:16 dist-test-slave-65tz krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:16.682622 12695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:16.683619 12695 log.cc:824] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:16.685503 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:16.671920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.8.221.254:35571)
0620 14:48:16.672480 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:16.672496 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:16.672532 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:48:16.673244 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:16.673261 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:16.673605 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:16.673977 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:16.674753 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.675756 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:16.676680 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.677138 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:16.677311 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:16.680347 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:16.680352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:16.680365 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:16.680392 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:16.682324 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:16.682912 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:16.682917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:16.682921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:16.683048 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:16.683339 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:16.683341 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:16.683342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:16.683736 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:16.683935 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:16.684255 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 14:48:16.684575 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":204,"threads_started":1}
I20260620 14:48:16.686072 12695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768: No bootstrap required, opened a new log
I20260620 14:48:16.689970 12695 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1852049c917409ca179acf6f83f9768" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 35571 } }
I20260620 14:48:16.690188 12695 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:16.690232 12695 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1852049c917409ca179acf6f83f9768, State: Initialized, Role: FOLLOWER
I20260620 14:48:16.690608 12695 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1852049c917409ca179acf6f83f9768" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 35571 } }
I20260620 14:48:16.690719 12695 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:16.690766 12695 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:16.690837 12695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:16.691890 12695 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1852049c917409ca179acf6f83f9768" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 35571 } }
I20260620 14:48:16.692258 12695 leader_election.cc:302] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1852049c917409ca179acf6f83f9768; no voters:
I20260620 14:48:16.692588 12695 leader_election.cc:288] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:16.692785 12700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:16.693125 12700 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768 [term 1 LEADER]: Becoming Leader. State: Replica: a1852049c917409ca179acf6f83f9768, State: Running, Role: LEADER
I20260620 14:48:16.693521 12700 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1852049c917409ca179acf6f83f9768" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 35571 } }
I20260620 14:48:16.693907 12695 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:16.695429 12701 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1852049c917409ca179acf6f83f9768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1852049c917409ca179acf6f83f9768" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 35571 } } }
I20260620 14:48:16.695572 12701 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:16.695528 12702 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1852049c917409ca179acf6f83f9768. Latest consensus state: current_term: 1 leader_uuid: "a1852049c917409ca179acf6f83f9768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1852049c917409ca179acf6f83f9768" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 35571 } } }
I20260620 14:48:16.695654 12702 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:16.696262 12712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:16.699725 12712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:16.706151 12712 catalog_manager.cc:1347] Generated new cluster ID: 28bbaec5465c48b594861fcc80e6e859
I20260620 14:48:16.706218 12712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:16.729370 12712 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:16.730868 12712 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:16.738724 12712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1852049c917409ca179acf6f83f9768: Generated new TSK 0
I20260620 14:48:16.739488 12712 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
I20260620 14:48:16.773132 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:35571
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:16.904537 12721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:16.904906 12721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:16.904958 12721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:16.909672 12721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:16.909776 12721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:16.909937 12721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:48:16.916123 12721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:35571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:16.917577 12721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:16.918941 12721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:16.930222 12726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:16.930274 12729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:16.930274 12727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:16.930516 12721 server_base.cc:1034] running on GCE node
I20260620 14:48:16.931126 12721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:16.931921 12721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:16.933101 12721 hybrid_clock.cc:648] HybridClock initialized: now 1781966896933068 us; error 50 us; skew 500 ppm
Jun 20 14:48:16 dist-test-slave-65tz krb5kdc[12614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966896, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:16.936304 12721 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:48:16.937425 12721 webserver.cc:460] Webserver started at http://127.8.221.193:39555/ using document root <none> and password file <none>
I20260620 14:48:16.938184 12721 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:16.938275 12721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:16.938541 12721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:16.940927 12721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "58a29ad58dc7426fbed6308fe992e108"
format_stamp: "Formatted at 2026-06-20 14:48:16 on dist-test-slave-65tz"
server_key: "794bcd5ea9525505bb2215b252834e35"
server_key_iv: "89e8ca3d87bed4291dbaaa68e382f1fb"
server_key_version: "encryptionkey@0"
I20260620 14:48:16.941574 12721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "58a29ad58dc7426fbed6308fe992e108"
format_stamp: "Formatted at 2026-06-20 14:48:16 on dist-test-slave-65tz"
server_key: "794bcd5ea9525505bb2215b252834e35"
server_key_iv: "89e8ca3d87bed4291dbaaa68e382f1fb"
server_key_version: "encryptionkey@0"
I20260620 14:48:16.946038 12721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:16.948962 12736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:16.950567 12721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:16.950722 12721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "58a29ad58dc7426fbed6308fe992e108"
format_stamp: "Formatted at 2026-06-20 14:48:16 on dist-test-slave-65tz"
server_key: "794bcd5ea9525505bb2215b252834e35"
server_key_iv: "89e8ca3d87bed4291dbaaa68e382f1fb"
server_key_version: "encryptionkey@0"
I20260620 14:48:16.950860 12721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:16.973294 12721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:16.977010 12721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:16.977232 12721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:16.977983 12721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:16.978968 12721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:16.979038 12721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:16.979112 12721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:16.979157 12721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:16.990300 12721 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:33321
I20260620 14:48:16.990324 12849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:33321 every 8 connection(s)
I20260620 14:48:16.991418 12721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:16.991909 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 12721
I20260620 14:48:16.992048 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:48:16.992374 9079 external_mini_cluster.cc:1428] Setting key 5361e77483787f2f91083f9878a9641f
Jun 20 14:48:17 dist-test-slave-65tz krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966896, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.194@KRBTEST.COM" created.
I20260620 14:48:17.007751 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:16.993570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:40257 (local address 127.8.221.254:35571)
0620 14:48:16.993734 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:16.993740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:16.995056 (+ 1316us) server_negotiation.cc:407] Connection header received
0620 14:48:16.996879 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:16.996883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:16.996937 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:16.997014 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:16.998538 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.999310 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:17.000467 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:17.000710 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:17.000819 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:17.003545 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:17.003549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:17.003552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:17.003587 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:17.005428 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:17.006090 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:17.006094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:17.006096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:17.006190 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:17.006533 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:17.006536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:17.006539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:17.006734 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:17.006843 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:17.007457 (+ 614us) server_negotiation.cc:299] Negotiation successful
0620 14:48:17.007600 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:48:17.008869 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:16.994124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:35571 (local address 127.8.221.193:40257)
0620 14:48:16.994840 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:16.994927 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:16.996644 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:16.997210 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:16.997218 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:16.997619 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:16.998365 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:16.998385 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.999442 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:16.999444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:17.000341 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:17.000349 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:17.000863 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:17.000867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:17.001046 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:17.001788 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:17.001823 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:17.003339 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:17.005585 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:17.005590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:17.005604 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:17.005968 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:17.006290 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:17.006293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:17.006295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:17.006435 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:17.006836 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:17.006843 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:17.007093 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 14:48:17.007348 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 14:48:17.007722 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":161,"threads_started":1}
I20260620 14:48:17.010277 12850 heartbeater.cc:347] Connected to a master server at 127.8.221.254:35571
I20260620 14:48:17.010599 12850 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:17.011260 12850 heartbeater.cc:511] Master 127.8.221.254:35571 requested a full tablet report, sending...
I20260620 14:48:17.013103 12659 ts_manager.cc:194] Registered new tserver with Master: 58a29ad58dc7426fbed6308fe992e108 (127.8.221.193:33321)
I20260620 14:48:17.014456 12659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:40257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
I20260620 14:48:17.022728 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:35571
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:17.151857 12855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:17.152174 12855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:17.152249 12855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:17.157195 12855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:17.157279 12855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:17.157385 12855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:48:17.163919 12855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:35571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:17.165443 12855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:17.166941 12855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:17.178736 12863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:17.178731 12860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:17.178731 12861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:17.179251 12855 server_base.cc:1034] running on GCE node
I20260620 14:48:17.179750 12855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:17.180547 12855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:17.181746 12855 hybrid_clock.cc:648] HybridClock initialized: now 1781966897181730 us; error 34 us; skew 500 ppm
Jun 20 14:48:17 dist-test-slave-65tz krb5kdc[12614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966897, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:17.185066 12855 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:48:17.186131 12855 webserver.cc:460] Webserver started at http://127.8.221.194:41135/ using document root <none> and password file <none>
I20260620 14:48:17.186765 12855 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:17.186829 12855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:17.187008 12855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:17.189167 12855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "0de66530e1384a07af636ed3df988295"
format_stamp: "Formatted at 2026-06-20 14:48:17 on dist-test-slave-65tz"
server_key: "6579891b1c368321de7cf7c903038924"
server_key_iv: "097c17c43f35d7292568e378c7a69d0e"
server_key_version: "encryptionkey@0"
I20260620 14:48:17.189708 12855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "0de66530e1384a07af636ed3df988295"
format_stamp: "Formatted at 2026-06-20 14:48:17 on dist-test-slave-65tz"
server_key: "6579891b1c368321de7cf7c903038924"
server_key_iv: "097c17c43f35d7292568e378c7a69d0e"
server_key_version: "encryptionkey@0"
I20260620 14:48:17.193784 12855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260620 14:48:17.197094 12870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:17.198755 12855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:48:17.198907 12855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "0de66530e1384a07af636ed3df988295"
format_stamp: "Formatted at 2026-06-20 14:48:17 on dist-test-slave-65tz"
server_key: "6579891b1c368321de7cf7c903038924"
server_key_iv: "097c17c43f35d7292568e378c7a69d0e"
server_key_version: "encryptionkey@0"
I20260620 14:48:17.199054 12855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:17.221272 12855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:17.224362 12855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:17.224558 12855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:17.225142 12855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:17.226037 12855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:17.226104 12855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:17.226169 12855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:17.226213 12855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:17.235822 12855 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:46759
I20260620 14:48:17.235857 12983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:46759 every 8 connection(s)
I20260620 14:48:17.236718 12855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:17.240103 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 12855
I20260620 14:48:17.240219 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:17.240545 9079 external_mini_cluster.cc:1428] Setting key 4f53a331361ca90bf456dde32929a30e
Jun 20 14:48:17 dist-test-slave-65tz krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966897, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.195@KRBTEST.COM; defaulting to no policy
I20260620 14:48:17.253420 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:17.239028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:39387 (local address 127.8.221.254:35571)
0620 14:48:17.239183 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:17.239187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:17.240094 (+ 907us) server_negotiation.cc:407] Connection header received
0620 14:48:17.242106 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:17.242113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:17.242182 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:17.242281 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:17.243771 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:17.244544 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:17.246198 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:17.246618 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:17.246750 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:17.249775 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:17.249778 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:17.249780 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:17.249885 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:17.251408 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:17.251925 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:17.251930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:17.251932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:17.252007 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:17.252336 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:17.252342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:17.252343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:17.252523 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:17.252627 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:17.253137 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 14:48:17.253253 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.195@KRBTEST.COM" created.
I20260620 14:48:17.254400 12986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:17.239375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:35571 (local address 127.8.221.194:39387)
0620 14:48:17.239917 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:17.239970 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:17.241826 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:17.242491 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:17.242499 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:17.243017 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:17.243623 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:17.243638 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:17.244687 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:17.244692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:17.246070 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:17.246080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:17.246784 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:17.246786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:17.246951 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:17.247678 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:17.247706 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:17.249598 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:17.251537 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:17.251543 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:17.251558 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:17.251825 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:17.252095 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:17.252099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:17.252101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:17.252246 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:17.252625 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:17.252632 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:17.252900 (+ 268us) client_negotiation.cc:769] Sending connection context
0620 14:48:17.253100 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 14:48:17.253443 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":119,"threads_started":1}
I20260620 14:48:17.255829 12984 heartbeater.cc:347] Connected to a master server at 127.8.221.254:35571
I20260620 14:48:17.256206 12984 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:17.256994 12984 heartbeater.cc:511] Master 127.8.221.254:35571 requested a full tablet report, sending...
I20260620 14:48:17.258298 12659 ts_manager.cc:194] Registered new tserver with Master: 0de66530e1384a07af636ed3df988295 (127.8.221.194:46759)
I20260620 14:48:17.258896 12659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:39387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
I20260620 14:48:17.272934 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:35571
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:43633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:17.403342 12989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:17.403683 12989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:17.403766 12989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:17.408181 12989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:17.408264 12989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:17.408365 12989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:48:17.414237 12989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:43633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:35571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:17.415607 12989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:17.416890 12989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:17.428344 12994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:17.428349 12997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:17.428349 12995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:17.428403 12989 server_base.cc:1034] running on GCE node
I20260620 14:48:17.429176 12989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:17.429972 12989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:17.431150 12989 hybrid_clock.cc:648] HybridClock initialized: now 1781966897431124 us; error 39 us; skew 500 ppm
Jun 20 14:48:17 dist-test-slave-65tz krb5kdc[12614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966897, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:17.434154 12989 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:48:17.435045 12989 webserver.cc:460] Webserver started at http://127.8.221.195:46223/ using document root <none> and password file <none>
I20260620 14:48:17.435595 12989 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:17.435657 12989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:17.435822 12989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:17.437922 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "ee1c06bb1a5245ae8d0150ebbfad9f8c"
format_stamp: "Formatted at 2026-06-20 14:48:17 on dist-test-slave-65tz"
server_key: "459182bdfbae11d054a347d1a25bd220"
server_key_iv: "114b87b22e170fb3ff2942faf5df94ae"
server_key_version: "encryptionkey@0"
I20260620 14:48:17.438454 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "ee1c06bb1a5245ae8d0150ebbfad9f8c"
format_stamp: "Formatted at 2026-06-20 14:48:17 on dist-test-slave-65tz"
server_key: "459182bdfbae11d054a347d1a25bd220"
server_key_iv: "114b87b22e170fb3ff2942faf5df94ae"
server_key_version: "encryptionkey@0"
I20260620 14:48:17.442485 12989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:17.445115 13004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:17.446558 12989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:17.446687 12989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "ee1c06bb1a5245ae8d0150ebbfad9f8c"
format_stamp: "Formatted at 2026-06-20 14:48:17 on dist-test-slave-65tz"
server_key: "459182bdfbae11d054a347d1a25bd220"
server_key_iv: "114b87b22e170fb3ff2942faf5df94ae"
server_key_version: "encryptionkey@0"
I20260620 14:48:17.446797 12989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:17.468951 12989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:17.472368 12989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:17.472579 12989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:17.473201 12989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:17.474092 12989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:17.474160 12989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:17.474234 12989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:17.474275 12989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:17.483979 12989 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:42003
I20260620 14:48:17.484035 13117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:42003 every 8 connection(s)
I20260620 14:48:17.485025 12989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:17.490160 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 12989
I20260620 14:48:17.490324 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:17.490623 9079 external_mini_cluster.cc:1428] Setting key 6fbba897d1843bfa7e896dfb8871f80a
Jun 20 14:48:17 dist-test-slave-65tz krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966897, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:17.500727 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:17.488067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:52509 (local address 127.8.221.254:35571)
0620 14:48:17.488248 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:17.488252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:17.489230 (+ 978us) server_negotiation.cc:407] Connection header received
0620 14:48:17.491129 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:17.491132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:17.491186 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:17.491268 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:17.492594 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:17.493208 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:17.494494 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:17.494754 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:17.494862 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:17.497339 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:17.497342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:17.497344 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:17.497366 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:17.498779 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:17.499268 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:17.499273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:17.499275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:17.499349 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:17.499696 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:17.499701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:17.499703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:17.499873 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:17.499969 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:17.500453 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 14:48:17.500566 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:48:17.501554 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:17.488476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:35571 (local address 127.8.221.195:52509)
0620 14:48:17.489047 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:17.489098 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:17.490881 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:17.491447 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:17.491455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:17.491921 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:17.492430 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:17.492443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:17.493328 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:17.493336 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:17.494382 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:17.494391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:17.494909 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:17.494913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:17.495146 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:17.495732 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:17.495748 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:17.497167 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:17.498905 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:17.498910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:17.498925 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:17.499171 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:17.499443 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:17.499446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:17.499448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:17.499604 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:17.499980 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:17.499988 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:17.500227 (+ 239us) client_negotiation.cc:769] Sending connection context
0620 14:48:17.500414 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 14:48:17.500736 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":155,"threads_started":1}
I20260620 14:48:17.502861 13118 heartbeater.cc:347] Connected to a master server at 127.8.221.254:35571
I20260620 14:48:17.503234 13118 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:17.504209 13118 heartbeater.cc:511] Master 127.8.221.254:35571 requested a full tablet report, sending...
I20260620 14:48:17.505389 12659 ts_manager.cc:194] Registered new tserver with Master: ee1c06bb1a5245ae8d0150ebbfad9f8c (127.8.221.195:42003)
I20260620 14:48:17.506021 12659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:52509
I20260620 14:48:17.516646 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:48:17 dist-test-slave-65tz krb5kdc[12614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:48:17 dist-test-slave-65tz krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.193@KRBTEST.COM
I20260620 14:48:17.546818 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:17.536111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45328 (local address 127.8.221.193:33321)
0620 14:48:17.536526 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:17.536534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:17.536565 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:48:17.536634 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:17.536640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:17.536902 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:17.537061 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:17.537715 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:17.538702 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:17.539705 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:17.540205 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:17.540316 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:17.543034 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:17.543043 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:17.543061 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:17.543087 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:17.544745 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:17.545179 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:17.545184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:17.545186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:17.545268 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:17.545591 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:17.545594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:17.545595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:17.545964 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:17.546145 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:17.546375 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 14:48:17.546573 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":145,"threads_started":1}
Jun 20 14:48:17 dist-test-slave-65tz krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:17.558573 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:17.550174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37218 (local address 127.8.221.254:35571)
0620 14:48:17.550339 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:17.550346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:17.550403 (+ 57us) server_negotiation.cc:407] Connection header received
0620 14:48:17.550540 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:17.550544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:17.550626 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:17.550711 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:17.551358 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:17.551854 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:17.552700 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:17.552916 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:17.553035 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:17.555512 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:17.555515 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:17.555517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:17.555537 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:17.556960 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:17.557317 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:17.557320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:17.557321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:17.557389 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:17.557612 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:17.557614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:17.557615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:17.557903 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:17.558044 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:17.558327 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 14:48:17.558442 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 14:48:17.559078 12659 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37218
I20260620 14:48:17.559901 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 12721
I20260620 14:48:17.566795 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 12855
I20260620 14:48:17.574235 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 12989
I20260620 14:48:17.580454 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 12627
2026-06-20T14:48:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2281 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:17 dist-test-slave-65tz krb5kdc[13130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:17 dist-test-slave-65tz krb5kdc[13130](info): set up 2 sockets
Jun 20 14:48:17 dist-test-slave-65tz krb5kdc[13130](info): commencing operation
krb5kdc: starting...
W20260620 14:48:18.635234 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:48:18 dist-test-slave-65tz krb5kdc[13130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:48:18.766644 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38583
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:38583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:18.884538 13143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:18.884810 13143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:18.884852 13143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:18.888763 13143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:18.888819 13143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:18.888839 13143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:18.888856 13143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:18.888871 13143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:18.893908 13143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:38583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:18.894868 13143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:18.895825 13143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:18.904472 13151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:18.904461 13148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:18.904592 13149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:18.904918 13143 server_base.cc:1034] running on GCE node
I20260620 14:48:18.905632 13143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:18.907025 13143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:18.908241 13143 hybrid_clock.cc:648] HybridClock initialized: now 1781966898908216 us; error 42 us; skew 500 ppm
Jun 20 14:48:18 dist-test-slave-65tz krb5kdc[13130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966898, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:18.911360 13143 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:48:18.912240 13143 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:18.912334 13143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:18.912583 13143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:18.915126 13143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "f6e3066debe54058a400cbd01995ad95"
format_stamp: "Formatted at 2026-06-20 14:48:18 on dist-test-slave-65tz"
server_key: "7ddc8117a490571ef946b649e8f9a11b"
server_key_iv: "bc3f15bc189fe0734c7ce4716a539062"
server_key_version: "encryptionkey@0"
I20260620 14:48:18.915753 13143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "f6e3066debe54058a400cbd01995ad95"
format_stamp: "Formatted at 2026-06-20 14:48:18 on dist-test-slave-65tz"
server_key: "7ddc8117a490571ef946b649e8f9a11b"
server_key_iv: "bc3f15bc189fe0734c7ce4716a539062"
server_key_version: "encryptionkey@0"
I20260620 14:48:18.920027 13143 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:48:18.922785 13157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:18.924285 13143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:18.924396 13143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "f6e3066debe54058a400cbd01995ad95"
format_stamp: "Formatted at 2026-06-20 14:48:18 on dist-test-slave-65tz"
server_key: "7ddc8117a490571ef946b649e8f9a11b"
server_key_iv: "bc3f15bc189fe0734c7ce4716a539062"
server_key_version: "encryptionkey@0"
I20260620 14:48:18.924489 13143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:18.940937 13143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:18.944453 13143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:18.944628 13143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:18.953169 13143 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:38583
I20260620 14:48:18.953534 13209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:38583 every 8 connection(s)
I20260620 14:48:18.954031 13143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:18.957120 13210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:18.962733 13210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95: Bootstrap starting.
I20260620 14:48:18.963694 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 13143
I20260620 14:48:18.963850 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:48:18.964149 9079 external_mini_cluster.cc:1428] Setting key 57f6ab3d8eba7d34d36c9c63c2d38b31
I20260620 14:48:18.965440 13210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:18.966243 13210 log.cc:824] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:18.968307 13210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95: No bootstrap required, opened a new log
I20260620 14:48:18.972621 13210 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e3066debe54058a400cbd01995ad95" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38583 } }
I20260620 14:48:18.972939 13210 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:18.973012 13210 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e3066debe54058a400cbd01995ad95, State: Initialized, Role: FOLLOWER
I20260620 14:48:18.973543 13210 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e3066debe54058a400cbd01995ad95" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38583 } }
I20260620 14:48:18.973675 13210 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:18.973763 13210 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:18.973892 13210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 14:48:18 dist-test-slave-65tz krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:18.974915 13210 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e3066debe54058a400cbd01995ad95" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38583 } }
I20260620 14:48:18.975286 13210 leader_election.cc:302] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6e3066debe54058a400cbd01995ad95; no voters:
I20260620 14:48:18.975613 13210 leader_election.cc:288] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:18.975785 13215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:18.976377 13215 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95 [term 1 LEADER]: Becoming Leader. State: Replica: f6e3066debe54058a400cbd01995ad95, State: Running, Role: LEADER
I20260620 14:48:18.976819 13215 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e3066debe54058a400cbd01995ad95" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38583 } }
I20260620 14:48:18.976958 13210 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:18.978909 13217 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6e3066debe54058a400cbd01995ad95. Latest consensus state: current_term: 1 leader_uuid: "f6e3066debe54058a400cbd01995ad95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e3066debe54058a400cbd01995ad95" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38583 } } }
I20260620 14:48:18.979039 13217 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:18.979365 13216 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6e3066debe54058a400cbd01995ad95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e3066debe54058a400cbd01995ad95" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38583 } } }
I20260620 14:48:18.979588 13216 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:18.981350 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:18.965514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41848 (local address 127.8.221.254:38583)
0620 14:48:18.966071 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:18.966090 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:18.966131 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:48:18.966947 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:18.966968 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:18.967364 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:18.967698 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:18.968633 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:18.969715 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:18.970959 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:18.971476 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:18.971606 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:18.974911 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:18.974919 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:18.974938 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:18.974976 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:18.977354 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:18.977962 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:18.977968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:18.977975 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:18.978137 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:18.978516 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:18.978521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:18.978523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:18.979135 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:18.979377 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:18.979944 (+ 567us) server_negotiation.cc:299] Negotiation successful
0620 14:48:18.980294 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":174,"threads_started":1}
W20260620 14:48:18.982620 13229 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:48:18.982712 13229 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:48:18.983700 13230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:18.984599 13230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:18.992141 13230 catalog_manager.cc:1347] Generated new cluster ID: 6a8c80eb28214c96a44744c3298ac763
I20260620 14:48:18.992234 13230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:19.013595 13230 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:19.015380 13230 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:19.034674 13230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95: Generated new TSK 0
I20260620 14:48:19.035754 13230 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
I20260620 14:48:19.077137 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38583
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:19.202153 13236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:19.202489 13236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:19.202555 13236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:19.206954 13236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:19.207028 13236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:19.207154 13236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:48:19.212957 13236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:19.214406 13236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:19.215662 13236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:19.226785 13244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:19.226789 13241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:19.226789 13242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:19.227353 13236 server_base.cc:1034] running on GCE node
I20260620 14:48:19.227830 13236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:19.228614 13236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:19.229828 13236 hybrid_clock.cc:648] HybridClock initialized: now 1781966899229742 us; error 64 us; skew 500 ppm
Jun 20 14:48:19 dist-test-slave-65tz krb5kdc[13130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966899, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:19.233173 13236 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:48:19.234217 13236 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:19.234290 13236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:19.234540 13236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:19.236961 13236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "784ed24c8eac4018be86fb9958f8cde4"
format_stamp: "Formatted at 2026-06-20 14:48:19 on dist-test-slave-65tz"
server_key: "b8f8497b25d15c0c07764718da55e46f"
server_key_iv: "0fc2e5d2ef7697ff08e2759289782b6f"
server_key_version: "encryptionkey@0"
I20260620 14:48:19.237604 13236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "784ed24c8eac4018be86fb9958f8cde4"
format_stamp: "Formatted at 2026-06-20 14:48:19 on dist-test-slave-65tz"
server_key: "b8f8497b25d15c0c07764718da55e46f"
server_key_iv: "0fc2e5d2ef7697ff08e2759289782b6f"
server_key_version: "encryptionkey@0"
I20260620 14:48:19.242038 13236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 14:48:19.245257 13250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:19.246940 13236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:48:19.247104 13236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "784ed24c8eac4018be86fb9958f8cde4"
format_stamp: "Formatted at 2026-06-20 14:48:19 on dist-test-slave-65tz"
server_key: "b8f8497b25d15c0c07764718da55e46f"
server_key_iv: "0fc2e5d2ef7697ff08e2759289782b6f"
server_key_version: "encryptionkey@0"
I20260620 14:48:19.247237 13236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:19.276749 13236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:19.280784 13236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:19.281018 13236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:19.281746 13236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:19.282744 13236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:19.282812 13236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:19.282886 13236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:19.282927 13236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:19.294332 13236 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:35507
I20260620 14:48:19.294364 13363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:35507 every 8 connection(s)
I20260620 14:48:19.295106 13236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:19.305400 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 13236
I20260620 14:48:19.305557 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:48:19.305953 9079 external_mini_cluster.cc:1428] Setting key 92d263510ffb76262d5c6d32f07fce45
Jun 20 14:48:19 dist-test-slave-65tz krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966899, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:19.311439 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:19.297197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:55465 (local address 127.8.221.254:38583)
0620 14:48:19.297398 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:19.297403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:19.298362 (+ 959us) server_negotiation.cc:407] Connection header received
0620 14:48:19.299811 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:19.299814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:19.299866 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:19.299966 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:19.301481 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.302058 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.303348 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.303603 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.303691 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:19.306892 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:19.306896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:19.306899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:19.306926 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:19.308942 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:19.309580 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:19.309586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:19.309588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:19.309693 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:19.310109 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:19.310113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:19.310115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:19.310341 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:19.310455 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:19.311075 (+ 620us) server_negotiation.cc:299] Negotiation successful
0620 14:48:19.311247 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 14:48:19.312690 13366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:19.297572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38583 (local address 127.8.221.193:55465)
0620 14:48:19.298167 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:19.298221 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:19.299554 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:19.300263 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:19.300272 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:19.300714 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:19.301326 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:19.301342 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.302210 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.302213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:19.303217 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:19.303226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.303755 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.303757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:19.303915 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:19.304687 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:19.304707 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:19.306667 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:19.309093 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:19.309099 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:19.309117 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:19.309444 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:19.309837 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:19.309840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:19.309846 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:19.309999 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:19.310449 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:19.310457 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:19.310782 (+ 325us) client_negotiation.cc:769] Sending connection context
0620 14:48:19.311007 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 14:48:19.311422 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":165,"threads_started":1}
I20260620 14:48:19.314260 13364 heartbeater.cc:347] Connected to a master server at 127.8.221.254:38583
I20260620 14:48:19.314550 13364 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:19.315090 13364 heartbeater.cc:511] Master 127.8.221.254:38583 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.221.194@KRBTEST.COM; defaulting to no policy
I20260620 14:48:19.316891 13174 ts_manager.cc:194] Registered new tserver with Master: 784ed24c8eac4018be86fb9958f8cde4 (127.8.221.193:35507)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.194@KRBTEST.COM" created.
I20260620 14:48:19.318360 13174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:55465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
I20260620 14:48:19.333513 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38583
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:19.452566 13369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:19.452833 13369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:19.452880 13369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:19.456780 13369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:19.456837 13369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:19.456897 13369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:48:19.462165 13369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:19.463179 13369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:19.464155 13369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:19.473865 13377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:19.473861 13375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:19.473904 13374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:19.474368 13369 server_base.cc:1034] running on GCE node
I20260620 14:48:19.474771 13369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:19.475416 13369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:19.476593 13369 hybrid_clock.cc:648] HybridClock initialized: now 1781966899476553 us; error 55 us; skew 500 ppm
Jun 20 14:48:19 dist-test-slave-65tz krb5kdc[13130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966899, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:19.479785 13369 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:48:19.480544 13369 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:19.480612 13369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:19.480827 13369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:19.482928 13369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "6a86552a52d640989f3f65cecfc7fa13"
format_stamp: "Formatted at 2026-06-20 14:48:19 on dist-test-slave-65tz"
server_key: "179be23a08a8952da2d34ae905bdb7c7"
server_key_iv: "33007e1d7e212b4e2685a3cea06b6f5b"
server_key_version: "encryptionkey@0"
I20260620 14:48:19.483486 13369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "6a86552a52d640989f3f65cecfc7fa13"
format_stamp: "Formatted at 2026-06-20 14:48:19 on dist-test-slave-65tz"
server_key: "179be23a08a8952da2d34ae905bdb7c7"
server_key_iv: "33007e1d7e212b4e2685a3cea06b6f5b"
server_key_version: "encryptionkey@0"
I20260620 14:48:19.487500 13369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:48:19.490293 13383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:19.491768 13369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:19.491914 13369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "6a86552a52d640989f3f65cecfc7fa13"
format_stamp: "Formatted at 2026-06-20 14:48:19 on dist-test-slave-65tz"
server_key: "179be23a08a8952da2d34ae905bdb7c7"
server_key_iv: "33007e1d7e212b4e2685a3cea06b6f5b"
server_key_version: "encryptionkey@0"
I20260620 14:48:19.492025 13369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:19.509963 13369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:19.512990 13369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:19.513191 13369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:19.513779 13369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:19.514786 13369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:19.514853 13369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:19.514918 13369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:19.514964 13369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:19.524900 13369 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:45869
I20260620 14:48:19.524968 13496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:45869 every 8 connection(s)
I20260620 14:48:19.525521 13369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:19.530407 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 13369
I20260620 14:48:19.530531 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:19.530902 9079 external_mini_cluster.cc:1428] Setting key 3db1c8102282bf0788f960c32f979ded
Jun 20 14:48:19 dist-test-slave-65tz krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966899, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.195@KRBTEST.COM; defaulting to no policy
I20260620 14:48:19.542563 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:19.527695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:45847 (local address 127.8.221.254:38583)
0620 14:48:19.527920 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:19.527927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:19.529071 (+ 1144us) server_negotiation.cc:407] Connection header received
0620 14:48:19.530703 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:19.530708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:19.530777 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:19.530895 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:19.532641 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.533424 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.535281 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.535509 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.535605 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:19.538877 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:19.538881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:19.538888 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:19.538910 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:19.540450 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:19.541023 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:19.541026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:19.541028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:19.541103 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:19.541446 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:19.541449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:19.541451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:19.541624 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:19.541737 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:19.542281 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 14:48:19.542398 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:48:19.543468 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:19.528126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38583 (local address 127.8.221.194:45847)
0620 14:48:19.528865 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:19.528932 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:19.530409 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:19.531097 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:19.531106 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:19.531649 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:19.532445 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:19.532469 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.533572 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.533574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:19.535123 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:19.535136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.535697 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.535704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:19.535943 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:19.536670 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:19.536698 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:19.538681 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:19.540592 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:19.540598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:19.540613 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:19.540917 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:19.541196 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:19.541199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:19.541200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:19.541348 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:19.541725 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:19.541732 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:19.542047 (+ 315us) client_negotiation.cc:769] Sending connection context
0620 14:48:19.542238 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 14:48:19.542579 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":158,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.195@KRBTEST.COM" created.
I20260620 14:48:19.545082 13497 heartbeater.cc:347] Connected to a master server at 127.8.221.254:38583
I20260620 14:48:19.545392 13497 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:19.546041 13497 heartbeater.cc:511] Master 127.8.221.254:38583 requested a full tablet report, sending...
I20260620 14:48:19.547398 13174 ts_manager.cc:194] Registered new tserver with Master: 6a86552a52d640989f3f65cecfc7fa13 (127.8.221.194:45869)
I20260620 14:48:19.548377 13174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:45847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
I20260620 14:48:19.562883 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38583
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:32927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:19.692387 13502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:19.692723 13502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:19.692773 13502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:19.696884 13502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:19.696945 13502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:19.697009 13502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:48:19.702490 13502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:32927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:19.703722 13502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:19.704936 13502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:19.715849 13507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:19.715849 13508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:19.715907 13510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:19.716048 13502 server_base.cc:1034] running on GCE node
I20260620 14:48:19.716737 13502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:19.717546 13502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:19.718715 13502 hybrid_clock.cc:648] HybridClock initialized: now 1781966899718692 us; error 42 us; skew 500 ppm
Jun 20 14:48:19 dist-test-slave-65tz krb5kdc[13130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966899, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:19.721872 13502 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:48:19.722666 13502 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:19.722762 13502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:19.722981 13502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:19.725112 13502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "7a7baf45f1f849d28859bb930a83eeeb"
format_stamp: "Formatted at 2026-06-20 14:48:19 on dist-test-slave-65tz"
server_key: "543f5ce9efa6977576c198cc2a0b8d6d"
server_key_iv: "d0422cca34bc929a1048154da57b6418"
server_key_version: "encryptionkey@0"
I20260620 14:48:19.725715 13502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "7a7baf45f1f849d28859bb930a83eeeb"
format_stamp: "Formatted at 2026-06-20 14:48:19 on dist-test-slave-65tz"
server_key: "543f5ce9efa6977576c198cc2a0b8d6d"
server_key_iv: "d0422cca34bc929a1048154da57b6418"
server_key_version: "encryptionkey@0"
I20260620 14:48:19.729789 13502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:19.732390 13516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:19.733974 13502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:19.734118 13502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "7a7baf45f1f849d28859bb930a83eeeb"
format_stamp: "Formatted at 2026-06-20 14:48:19 on dist-test-slave-65tz"
server_key: "543f5ce9efa6977576c198cc2a0b8d6d"
server_key_iv: "d0422cca34bc929a1048154da57b6418"
server_key_version: "encryptionkey@0"
I20260620 14:48:19.734226 13502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:19.757164 13502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:19.760226 13502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:19.760413 13502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:19.760979 13502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:19.761834 13502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:19.761904 13502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:19.761970 13502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:19.762010 13502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:19.771579 13502 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:33915
I20260620 14:48:19.771625 13629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:33915 every 8 connection(s)
I20260620 14:48:19.772148 13502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:19.781759 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 13502
I20260620 14:48:19.781944 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:19.782353 9079 external_mini_cluster.cc:1428] Setting key 7e1576c3c58cbd5f5cebb2e60021a747
Jun 20 14:48:19 dist-test-slave-65tz krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966899, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:19.790814 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:19.774226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:35541 (local address 127.8.221.254:38583)
0620 14:48:19.774418 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:19.774423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:19.775272 (+ 849us) server_negotiation.cc:407] Connection header received
0620 14:48:19.776741 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:19.776744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:19.776792 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:19.776896 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:19.779091 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.779990 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.781926 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.782153 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.782264 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:19.785460 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:19.785465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:19.785468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:19.785502 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:19.787838 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:19.788768 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:19.788771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:19.788772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:19.788873 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:19.789422 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:19.789428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:19.789432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:19.789684 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:19.789829 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:19.790512 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 14:48:19.790643 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 14:48:19.791862 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:19.774554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38583 (local address 127.8.221.195:35541)
0620 14:48:19.775055 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:19.775117 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:19.776509 (+ 1392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:19.777103 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:19.777116 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:19.777718 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:19.778899 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:19.778927 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.780163 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.780169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:19.781725 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:19.781739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.782301 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.782305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:19.782536 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:19.783198 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:19.783220 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:19.785269 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:19.787979 (+ 2710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:19.787988 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:19.788027 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:19.788418 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:19.788995 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:19.788999 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:19.789001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:19.789303 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:19.789899 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:19.789906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:19.790217 (+ 311us) client_negotiation.cc:769] Sending connection context
0620 14:48:19.790466 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 14:48:19.790797 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":116,"threads_started":1}
I20260620 14:48:19.793751 13630 heartbeater.cc:347] Connected to a master server at 127.8.221.254:38583
I20260620 14:48:19.794072 13630 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:19.794660 13630 heartbeater.cc:511] Master 127.8.221.254:38583 requested a full tablet report, sending...
I20260620 14:48:19.796115 13174 ts_manager.cc:194] Registered new tserver with Master: 7a7baf45f1f849d28859bb930a83eeeb (127.8.221.195:33915)
I20260620 14:48:19.797078 13174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:35541
I20260620 14:48:19.798823 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:48:19.811614 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:19.801092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.8.221.254:38583)
0620 14:48:19.801227 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:19.801232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:19.801604 (+ 372us) server_negotiation.cc:407] Connection header received
0620 14:48:19.801746 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:19.801753 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:19.801846 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:19.801936 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:19.803008 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.803855 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.805205 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.805488 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.805617 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:19.807636 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:19.807639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:19.807642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:19.807669 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:19.809536 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:19.810073 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:19.810077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:19.810080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:19.810180 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:19.810516 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:19.810519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:19.810521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:19.810863 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:19.811025 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:19.811283 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 14:48:19.811438 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:48:19.815040 13174 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:48:19.818584 13174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:19.834939 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:19.828214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:45869 (local address 127.0.0.1:45892)
0620 14:48:19.828686 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:19.828746 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:19.828883 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:19.830089 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:19.830097 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:19.830127 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:19.830275 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:19.830282 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.831735 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.831742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:19.833055 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:19.833068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.834239 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.834243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:19.834380 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:19.834445 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 14:48:19.834561 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 14:48:19.834737 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":6,"thread_start_us":102,"threads_started":1}
I20260620 14:48:19.835301 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:19.828597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:33915 (local address 127.0.0.1:55798)
0620 14:48:19.829829 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:19.829844 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:19.829992 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:19.830720 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:19.830725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:19.830749 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:19.830890 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:19.830898 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.832318 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.832322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:19.833431 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:19.833437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.834556 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.834560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:19.834744 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:19.834768 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:48:19.834877 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:48:19.834986 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1128,"mutex_wait_us":53,"thread_start_us":143,"threads_started":1}
I20260620 14:48:19.835331 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:19.829184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55798 (local address 127.8.221.195:33915)
0620 14:48:19.829647 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:19.829662 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:19.829931 (+ 269us) server_negotiation.cc:407] Connection header received
0620 14:48:19.830150 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:19.830156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:19.830365 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:19.830493 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:19.831043 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.832155 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.833567 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.834434 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.834553 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:19.834680 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:19.834909 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 14:48:19.835003 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":146,"threads_started":1}
I20260620 14:48:19.835338 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:19.829248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.8.221.193:35507)
0620 14:48:19.829563 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:19.829569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:19.830075 (+ 506us) server_negotiation.cc:407] Connection header received
0620 14:48:19.830158 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:19.830163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:19.830365 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:19.830517 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:19.830770 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.831689 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.833593 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.834344 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.834447 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:19.834607 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:19.835032 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 14:48:19.835165 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":83,"threads_started":1}
I20260620 14:48:19.835708 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:19.828404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45892 (local address 127.8.221.194:45869)
0620 14:48:19.829210 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:19.829226 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:19.829256 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:48:19.829372 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:19.829379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:19.829623 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:19.829864 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:19.830391 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.831581 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.833213 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.834098 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.834243 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:19.834412 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:19.835375 (+ 963us) server_negotiation.cc:299] Negotiation successful
0620 14:48:19.835504 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":161,"threads_started":1}
I20260620 14:48:19.836069 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:19.829082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:35507 (local address 127.0.0.1:42430)
0620 14:48:19.829811 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:19.829828 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:19.829958 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:19.830510 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:19.830513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:19.830531 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:19.830618 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:19.830624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.831829 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.831833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:19.833423 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:19.833432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.834666 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.834669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:19.834790 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:19.834811 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:48:19.834879 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:48:19.834945 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":14,"thread_start_us":125,"threads_started":1}
I20260620 14:48:19.838969 13431 tablet_service.cc:1467] Processing CreateTablet for tablet eb8205476190409298381ad4b158eb32 (DEFAULT_TABLE table=test-table [id=a4b578a95911411eaf56e88151a2762a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:19.839300 13298 tablet_service.cc:1467] Processing CreateTablet for tablet eb8205476190409298381ad4b158eb32 (DEFAULT_TABLE table=test-table [id=a4b578a95911411eaf56e88151a2762a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:19.839303 13564 tablet_service.cc:1467] Processing CreateTablet for tablet eb8205476190409298381ad4b158eb32 (DEFAULT_TABLE table=test-table [id=a4b578a95911411eaf56e88151a2762a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:19.840129 13431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb8205476190409298381ad4b158eb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:19.840469 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb8205476190409298381ad4b158eb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:19.840478 13564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb8205476190409298381ad4b158eb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:19.846808 13647 tablet_bootstrap.cc:492] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13: Bootstrap starting.
I20260620 14:48:19.848487 13648 tablet_bootstrap.cc:492] T eb8205476190409298381ad4b158eb32 P 784ed24c8eac4018be86fb9958f8cde4: Bootstrap starting.
I20260620 14:48:19.849460 13649 tablet_bootstrap.cc:492] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb: Bootstrap starting.
I20260620 14:48:19.849788 13647 tablet_bootstrap.cc:654] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:19.850728 13647 log.cc:824] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:19.850854 13648 tablet_bootstrap.cc:654] T eb8205476190409298381ad4b158eb32 P 784ed24c8eac4018be86fb9958f8cde4: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:19.851636 13648 log.cc:824] T eb8205476190409298381ad4b158eb32 P 784ed24c8eac4018be86fb9958f8cde4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:19.851888 13649 tablet_bootstrap.cc:654] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:19.853245 13649 log.cc:824] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:19.853433 13647 tablet_bootstrap.cc:492] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13: No bootstrap required, opened a new log
I20260620 14:48:19.853595 13648 tablet_bootstrap.cc:492] T eb8205476190409298381ad4b158eb32 P 784ed24c8eac4018be86fb9958f8cde4: No bootstrap required, opened a new log
I20260620 14:48:19.853646 13647 ts_tablet_manager.cc:1397] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 14:48:19.853763 13648 ts_tablet_manager.cc:1397] T eb8205476190409298381ad4b158eb32 P 784ed24c8eac4018be86fb9958f8cde4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:48:19.855355 13649 tablet_bootstrap.cc:492] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb: No bootstrap required, opened a new log
I20260620 14:48:19.855579 13649 ts_tablet_manager.cc:1397] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:48:19.857985 13647 raft_consensus.cc:348] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86552a52d640989f3f65cecfc7fa13" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45869 } } peers { permanent_uuid: "784ed24c8eac4018be86fb9958f8cde4" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35507 } } peers { permanent_uuid: "7a7baf45f1f849d28859bb930a83eeeb" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33915 } }
I20260620 14:48:19.858294 13647 raft_consensus.cc:374] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:19.858353 13647 raft_consensus.cc:729] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a86552a52d640989f3f65cecfc7fa13, State: Initialized, Role: FOLLOWER
I20260620 14:48:19.858294 13648 raft_consensus.cc:348] T eb8205476190409298381ad4b158eb32 P 784ed24c8eac4018be86fb9958f8cde4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86552a52d640989f3f65cecfc7fa13" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45869 } } peers { permanent_uuid: "784ed24c8eac4018be86fb9958f8cde4" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35507 } } peers { permanent_uuid: "7a7baf45f1f849d28859bb930a83eeeb" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33915 } }
I20260620 14:48:19.858515 13648 raft_consensus.cc:374] T eb8205476190409298381ad4b158eb32 P 784ed24c8eac4018be86fb9958f8cde4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:19.858564 13648 raft_consensus.cc:729] T eb8205476190409298381ad4b158eb32 P 784ed24c8eac4018be86fb9958f8cde4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 784ed24c8eac4018be86fb9958f8cde4, State: Initialized, Role: FOLLOWER
I20260620 14:48:19.858876 13647 consensus_queue.cc:260] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86552a52d640989f3f65cecfc7fa13" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45869 } } peers { permanent_uuid: "784ed24c8eac4018be86fb9958f8cde4" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35507 } } peers { permanent_uuid: "7a7baf45f1f849d28859bb930a83eeeb" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33915 } }
I20260620 14:48:19.858911 13648 consensus_queue.cc:260] T eb8205476190409298381ad4b158eb32 P 784ed24c8eac4018be86fb9958f8cde4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86552a52d640989f3f65cecfc7fa13" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45869 } } peers { permanent_uuid: "784ed24c8eac4018be86fb9958f8cde4" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35507 } } peers { permanent_uuid: "7a7baf45f1f849d28859bb930a83eeeb" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33915 } }
I20260620 14:48:19.859180 13649 raft_consensus.cc:348] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86552a52d640989f3f65cecfc7fa13" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45869 } } peers { permanent_uuid: "784ed24c8eac4018be86fb9958f8cde4" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35507 } } peers { permanent_uuid: "7a7baf45f1f849d28859bb930a83eeeb" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33915 } }
I20260620 14:48:19.859488 13649 raft_consensus.cc:374] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:19.859549 13649 raft_consensus.cc:729] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7baf45f1f849d28859bb930a83eeeb, State: Initialized, Role: FOLLOWER
I20260620 14:48:19.859922 13647 ts_tablet_manager.cc:1428] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260620 14:48:19.859922 13648 ts_tablet_manager.cc:1428] T eb8205476190409298381ad4b158eb32 P 784ed24c8eac4018be86fb9958f8cde4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:48:19.860385 13497 heartbeater.cc:503] Master 127.8.221.254:38583 was elected leader, sending a full tablet report...
I20260620 14:48:19.860512 13364 heartbeater.cc:503] Master 127.8.221.254:38583 was elected leader, sending a full tablet report...
I20260620 14:48:19.860091 13649 consensus_queue.cc:260] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86552a52d640989f3f65cecfc7fa13" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45869 } } peers { permanent_uuid: "784ed24c8eac4018be86fb9958f8cde4" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35507 } } peers { permanent_uuid: "7a7baf45f1f849d28859bb930a83eeeb" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33915 } }
I20260620 14:48:19.862602 13649 ts_tablet_manager.cc:1428] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 14:48:19.863003 13630 heartbeater.cc:503] Master 127.8.221.254:38583 was elected leader, sending a full tablet report...
I20260620 14:48:19.941390 13654 raft_consensus.cc:482] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:19.941713 13654 raft_consensus.cc:504] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86552a52d640989f3f65cecfc7fa13" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45869 } } peers { permanent_uuid: "784ed24c8eac4018be86fb9958f8cde4" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35507 } } peers { permanent_uuid: "7a7baf45f1f849d28859bb930a83eeeb" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33915 } }
I20260620 14:48:19.943223 13654 leader_election.cc:288] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 784ed24c8eac4018be86fb9958f8cde4 (127.8.221.193:35507), 7a7baf45f1f849d28859bb930a83eeeb (127.8.221.195:33915)
I20260620 14:48:19.947611 13499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:19.943614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:33915 (local address 127.8.221.194:47749)
0620 14:48:19.943766 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:19.943786 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:19.943900 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:19.944196 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:19.944199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:19.944251 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:19.944410 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:19.944417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.945483 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.945489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:19.946456 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:19.946465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.947122 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.947126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:19.947242 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:19.947270 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:48:19.947356 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:48:19.947434 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":41}
I20260620 14:48:19.947575 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:19.943712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:47749 (local address 127.8.221.195:33915)
0620 14:48:19.943872 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:19.943878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:19.943899 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:48:19.944003 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:19.944006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:19.944062 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:19.944157 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:19.944663 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.945209 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.946584 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.946991 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.947122 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:19.947174 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:19.947371 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 14:48:19.947434 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:48:19.947901 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:19.943721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:37853 (local address 127.8.221.193:35507)
0620 14:48:19.943863 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:19.943867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:19.944582 (+ 715us) server_negotiation.cc:407] Connection header received
0620 14:48:19.944726 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:19.944728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:19.944774 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:19.944857 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:19.945119 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.945670 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.946818 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.947250 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.947382 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:19.947450 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:19.947659 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 14:48:19.947752 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:48:19.948343 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:19.943614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:35507 (local address 127.8.221.194:37853)
0620 14:48:19.944444 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:19.944462 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:19.944581 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:19.944865 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:19.944868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:19.944883 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:19.945011 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:19.945017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.945821 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.945826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:19.946693 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:19.946701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.947374 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.947378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:19.947485 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:19.947513 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:48:19.948150 (+ 637us) client_negotiation.cc:240] Negotiation successful
0620 14:48:19.948228 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":375,"thread_start_us":174,"threads_started":1}
I20260620 14:48:19.948319 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8205476190409298381ad4b158eb32" candidate_uuid: "6a86552a52d640989f3f65cecfc7fa13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7baf45f1f849d28859bb930a83eeeb" is_pre_election: true
I20260620 14:48:19.948647 13584 raft_consensus.cc:2454] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a86552a52d640989f3f65cecfc7fa13 in term 0.
I20260620 14:48:19.949175 13386 leader_election.cc:302] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a86552a52d640989f3f65cecfc7fa13, 7a7baf45f1f849d28859bb930a83eeeb; no voters:
I20260620 14:48:19.949163 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8205476190409298381ad4b158eb32" candidate_uuid: "6a86552a52d640989f3f65cecfc7fa13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "784ed24c8eac4018be86fb9958f8cde4" is_pre_election: true
I20260620 14:48:19.949476 13654 raft_consensus.cc:2781] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:19.949501 13318 raft_consensus.cc:2454] T eb8205476190409298381ad4b158eb32 P 784ed24c8eac4018be86fb9958f8cde4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a86552a52d640989f3f65cecfc7fa13 in term 0.
I20260620 14:48:19.949561 13654 raft_consensus.cc:482] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:19.949599 13654 raft_consensus.cc:3037] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:19.950766 13654 raft_consensus.cc:504] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86552a52d640989f3f65cecfc7fa13" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45869 } } peers { permanent_uuid: "784ed24c8eac4018be86fb9958f8cde4" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35507 } } peers { permanent_uuid: "7a7baf45f1f849d28859bb930a83eeeb" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33915 } }
I20260620 14:48:19.951205 13654 leader_election.cc:288] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [CANDIDATE]: Term 1 election: Requested vote from peers 784ed24c8eac4018be86fb9958f8cde4 (127.8.221.193:35507), 7a7baf45f1f849d28859bb930a83eeeb (127.8.221.195:33915)
I20260620 14:48:19.951634 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8205476190409298381ad4b158eb32" candidate_uuid: "6a86552a52d640989f3f65cecfc7fa13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "784ed24c8eac4018be86fb9958f8cde4"
I20260620 14:48:19.951685 13584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8205476190409298381ad4b158eb32" candidate_uuid: "6a86552a52d640989f3f65cecfc7fa13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a7baf45f1f849d28859bb930a83eeeb"
I20260620 14:48:19.951823 13318 raft_consensus.cc:3037] T eb8205476190409298381ad4b158eb32 P 784ed24c8eac4018be86fb9958f8cde4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:19.951841 13584 raft_consensus.cc:3037] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:19.953106 13318 raft_consensus.cc:2454] T eb8205476190409298381ad4b158eb32 P 784ed24c8eac4018be86fb9958f8cde4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a86552a52d640989f3f65cecfc7fa13 in term 1.
I20260620 14:48:19.953436 13584 raft_consensus.cc:2454] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a86552a52d640989f3f65cecfc7fa13 in term 1.
I20260620 14:48:19.953467 13385 leader_election.cc:302] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a86552a52d640989f3f65cecfc7fa13, 784ed24c8eac4018be86fb9958f8cde4; no voters:
I20260620 14:48:19.953689 13654 raft_consensus.cc:2781] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:19.953979 13654 raft_consensus.cc:686] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [term 1 LEADER]: Becoming Leader. State: Replica: 6a86552a52d640989f3f65cecfc7fa13, State: Running, Role: LEADER
I20260620 14:48:19.954360 13654 consensus_queue.cc:237] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86552a52d640989f3f65cecfc7fa13" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45869 } } peers { permanent_uuid: "784ed24c8eac4018be86fb9958f8cde4" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35507 } } peers { permanent_uuid: "7a7baf45f1f849d28859bb930a83eeeb" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33915 } }
I20260620 14:48:19.957415 13174 catalog_manager.cc:5495] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a86552a52d640989f3f65cecfc7fa13 (127.8.221.194). New cstate: current_term: 1 leader_uuid: "6a86552a52d640989f3f65cecfc7fa13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a86552a52d640989f3f65cecfc7fa13" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "784ed24c8eac4018be86fb9958f8cde4" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a7baf45f1f849d28859bb930a83eeeb" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33915 } health_report { overall_health: UNKNOWN } } }
I20260620 14:48:19.990640 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:19.987034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45900 (local address 127.8.221.194:45869)
0620 14:48:19.987206 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:19.987211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:19.987243 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:48:19.987332 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:19.987334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:19.987386 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:19.987495 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:19.987795 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.988312 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.989133 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:19.989347 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:19.989451 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:19.989753 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:19.990213 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:19.990352 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:19.990415 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 14:48:19.990471 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:48:19.998369 13584 raft_consensus.cc:1261] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb [term 1 FOLLOWER]: Refusing update from remote peer 6a86552a52d640989f3f65cecfc7fa13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:19.998380 13318 raft_consensus.cc:1261] T eb8205476190409298381ad4b158eb32 P 784ed24c8eac4018be86fb9958f8cde4 [term 1 FOLLOWER]: Refusing update from remote peer 6a86552a52d640989f3f65cecfc7fa13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:19.999162 13657 consensus_queue.cc:1035] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "784ed24c8eac4018be86fb9958f8cde4" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:19.999423 13654 consensus_queue.cc:1035] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a7baf45f1f849d28859bb930a83eeeb" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:20.007045 13660 mvcc.cc:205] Tried to move back new op lower bound from 7298936422384312320 to 7298936422215819264. Current Snapshot: MvccSnapshot[applied={T|T < 7298936422384312320}]
I20260620 14:48:20.009227 13662 mvcc.cc:205] Tried to move back new op lower bound from 7298936422384312320 to 7298936422215819264. Current Snapshot: MvccSnapshot[applied={T|T < 7298936422384312320}]
I20260620 14:48:20.018181 13173 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:20.018402 13173 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:20.021443 13173 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6e3066debe54058a400cbd01995ad95: Sending DeleteTablet for 3 replicas of tablet eb8205476190409298381ad4b158eb32
I20260620 14:48:20.022531 13564 tablet_service.cc:1514] Processing DeleteTablet for tablet eb8205476190409298381ad4b158eb32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:20 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:55798
I20260620 14:48:20.022621 13298 tablet_service.cc:1514] Processing DeleteTablet for tablet eb8205476190409298381ad4b158eb32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:20 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:42430
I20260620 14:48:20.023164 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 13236
I20260620 14:48:20.023138 13431 tablet_service.cc:1514] Processing DeleteTablet for tablet eb8205476190409298381ad4b158eb32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:20 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:45892
I20260620 14:48:20.024091 13669 tablet_replica.cc:330] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb: stopping tablet replica
I20260620 14:48:20.024593 13669 raft_consensus.cc:2229] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:20.025066 13669 raft_consensus.cc:2258] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:20.025668 13671 tablet_replica.cc:330] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13: stopping tablet replica
I20260620 14:48:20.026055 13671 raft_consensus.cc:2229] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:20.026455 13671 raft_consensus.cc:2258] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:20.027220 13669 ts_tablet_manager.cc:1905] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:20.027856 13671 ts_tablet_manager.cc:1905] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:20.030889 13671 ts_tablet_manager.cc:1918] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:20.030990 13671 log.cc:1196] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/eb8205476190409298381ad4b158eb32
I20260620 14:48:20.031313 13671 ts_tablet_manager.cc:1939] T eb8205476190409298381ad4b158eb32 P 6a86552a52d640989f3f65cecfc7fa13: Deleting consensus metadata
I20260620 14:48:20.031750 13669 ts_tablet_manager.cc:1918] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:20.031935 13669 log.cc:1196] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/eb8205476190409298381ad4b158eb32
I20260620 14:48:20.032542 13159 catalog_manager.cc:4841] TS 6a86552a52d640989f3f65cecfc7fa13 (127.8.221.194:45869): tablet eb8205476190409298381ad4b158eb32 (table test-table [id=a4b578a95911411eaf56e88151a2762a]) successfully deleted
I20260620 14:48:20.032370 13669 ts_tablet_manager.cc:1939] T eb8205476190409298381ad4b158eb32 P 7a7baf45f1f849d28859bb930a83eeeb: Deleting consensus metadata
W20260620 14:48:20.033309 13159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.221.193:35507 (error 108)
I20260620 14:48:20.033643 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 13369
I20260620 14:48:20.036091 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:20.035793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:35507 (local address 127.0.0.1:42440)
0620 14:48:20.035925 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:20.035998 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.193:35507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 14:48:20.036279 13159 catalog_manager.cc:4584] TS 784ed24c8eac4018be86fb9958f8cde4 (127.8.221.193:35507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb8205476190409298381ad4b158eb32: Network error: Client connection negotiation failed: client connection to 127.8.221.193:35507: connect: Connection refused (error 111)
I20260620 14:48:20.036914 13160 catalog_manager.cc:4841] TS 7a7baf45f1f849d28859bb930a83eeeb (127.8.221.195:33915): tablet eb8205476190409298381ad4b158eb32 (table test-table [id=a4b578a95911411eaf56e88151a2762a]) successfully deleted
I20260620 14:48:20.041503 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 13502
I20260620 14:48:20.048343 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 13143
2026-06-20T14:48:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2467 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T14:48:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:20Z Disabled control of system clock
I20260620 14:48:20.082458 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:33137
--webserver_interface=127.8.221.254
--webserver_port=0
--builtin_ntp_servers=127.8.221.212:44375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:33137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:48:20.211181 13676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:20.211496 13676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:20.211537 13676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:20.215929 13676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:20.216027 13676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:20.216053 13676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:20.216071 13676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:20.216118 13676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:20.221185 13676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:33137
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:33137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:20.222656 13676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:20.224385 13676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:20.234170 13684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:20.234151 13682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:20.234535 13676 server_base.cc:1034] running on GCE node
W20260620 14:48:20.234151 13681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:20.235162 13676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:20.236337 13676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:20.237545 13676 hybrid_clock.cc:648] HybridClock initialized: now 1781966900237532 us; error 35 us; skew 500 ppm
I20260620 14:48:20.242246 13676 webserver.cc:460] Webserver started at http://127.8.221.254:40321/ using document root <none> and password file <none>
I20260620 14:48:20.242872 13676 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:20.242966 13676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:20.243216 13676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:20.245325 13676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "40ec534bdca5490b818af016775ece0a"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-65tz"
server_key: "6317fc5492d111a38b7fd1cc8fd8081c"
server_key_iv: "11439db44d3a7eb01474e78e05950378"
server_key_version: "encryptionkey@0"
I20260620 14:48:20.245950 13676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "40ec534bdca5490b818af016775ece0a"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-65tz"
server_key: "6317fc5492d111a38b7fd1cc8fd8081c"
server_key_iv: "11439db44d3a7eb01474e78e05950378"
server_key_version: "encryptionkey@0"
I20260620 14:48:20.249989 13676 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:20.252621 13690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:20.254164 13676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:20.254312 13676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "40ec534bdca5490b818af016775ece0a"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-65tz"
server_key: "6317fc5492d111a38b7fd1cc8fd8081c"
server_key_iv: "11439db44d3a7eb01474e78e05950378"
server_key_version: "encryptionkey@0"
I20260620 14:48:20.254424 13676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:20.259008 13676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:20.259610 13676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:20.259793 13676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:20.267431 13676 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:33137
I20260620 14:48:20.267495 13742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:33137 every 8 connection(s)
I20260620 14:48:20.268435 13676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:20.269629 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 13676
I20260620 14:48:20.269769 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:48:20.270170 9079 external_mini_cluster.cc:1428] Setting key 493dd67eb8fb3b89a155fbe6a5f22236
I20260620 14:48:20.274098 13743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:20.276815 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:20.271802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.8.221.254:33137)
0620 14:48:20.273037 (+ 1235us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:20.273054 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:20.273093 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:48:20.273896 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:20.273916 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:20.274350 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:20.274709 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:20.275550 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:20.275556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:20.275568 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:20.275578 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:20.275632 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:20.275648 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:20.275802 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:20.275875 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 14:48:20.276114 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812,"thread_start_us":626,"threads_started":1}
I20260620 14:48:20.281234 13743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a: Bootstrap starting.
I20260620 14:48:20.283999 13743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:20.284852 13743 log.cc:824] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:20.287237 13743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a: No bootstrap required, opened a new log
I20260620 14:48:20.290972 13743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ec534bdca5490b818af016775ece0a" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33137 } }
I20260620 14:48:20.291224 13743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:20.291309 13743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40ec534bdca5490b818af016775ece0a, State: Initialized, Role: FOLLOWER
I20260620 14:48:20.291795 13743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ec534bdca5490b818af016775ece0a" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33137 } }
I20260620 14:48:20.291942 13743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:20.292025 13743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:20.292135 13743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:20.293118 13743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ec534bdca5490b818af016775ece0a" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33137 } }
I20260620 14:48:20.293473 13743 leader_election.cc:302] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40ec534bdca5490b818af016775ece0a; no voters:
I20260620 14:48:20.293852 13743 leader_election.cc:288] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:20.293927 13748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:20.294149 13748 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a [term 1 LEADER]: Becoming Leader. State: Replica: 40ec534bdca5490b818af016775ece0a, State: Running, Role: LEADER
I20260620 14:48:20.294453 13748 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ec534bdca5490b818af016775ece0a" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33137 } }
I20260620 14:48:20.295250 13743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:20.296474 13750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40ec534bdca5490b818af016775ece0a. Latest consensus state: current_term: 1 leader_uuid: "40ec534bdca5490b818af016775ece0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ec534bdca5490b818af016775ece0a" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33137 } } }
I20260620 14:48:20.296423 13749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40ec534bdca5490b818af016775ece0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ec534bdca5490b818af016775ece0a" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33137 } } }
I20260620 14:48:20.296600 13750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:20.296609 13749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:20.297772 13757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:20.301044 13757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:20.306092 13757 catalog_manager.cc:1347] Generated new cluster ID: 59a0bbe19ed746e89a11fa3f6a4bce4d
I20260620 14:48:20.306157 13757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:20.348726 13757 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:20.349071 13757 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:20.362895 13757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a: Generated new TSK 0
I20260620 14:48:20.364010 13757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:48:20.372956 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33137
--builtin_ntp_servers=127.8.221.212:44375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:48:20.489652 13767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:20.489931 13767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:20.489971 13767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:20.493957 13767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:20.494010 13767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:20.494064 13767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:48:20.499022 13767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:20.500064 13767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:20.501432 13767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:20.511402 13772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:20.511406 13773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:20.511742 13767 server_base.cc:1034] running on GCE node
W20260620 14:48:20.511761 13775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:20.512225 13767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:20.513028 13767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:20.514228 13767 hybrid_clock.cc:648] HybridClock initialized: now 1781966900514188 us; error 49 us; skew 500 ppm
I20260620 14:48:20.516469 13767 webserver.cc:460] Webserver started at http://127.8.221.193:39953/ using document root <none> and password file <none>
I20260620 14:48:20.517151 13767 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:20.517253 13767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:20.517524 13767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:20.519912 13767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "06dd2c627f8f4838ab6f2fda42e19aaf"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-65tz"
server_key: "44447084f9fdf010f58103a282b52b2d"
server_key_iv: "760f02a5b1d2d59cf0348ac294e5a5a4"
server_key_version: "encryptionkey@0"
I20260620 14:48:20.520545 13767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "06dd2c627f8f4838ab6f2fda42e19aaf"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-65tz"
server_key: "44447084f9fdf010f58103a282b52b2d"
server_key_iv: "760f02a5b1d2d59cf0348ac294e5a5a4"
server_key_version: "encryptionkey@0"
I20260620 14:48:20.524992 13767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:48:20.527868 13781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:20.529234 13767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:20.529372 13767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "06dd2c627f8f4838ab6f2fda42e19aaf"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-65tz"
server_key: "44447084f9fdf010f58103a282b52b2d"
server_key_iv: "760f02a5b1d2d59cf0348ac294e5a5a4"
server_key_version: "encryptionkey@0"
I20260620 14:48:20.529495 13767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:20.534686 13767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:20.535413 13767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:20.535619 13767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:20.536242 13767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:20.537281 13767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:20.537351 13767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:20.537431 13767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:20.537484 13767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:20.547824 13767 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:40049
I20260620 14:48:20.547919 13894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:40049 every 8 connection(s)
I20260620 14:48:20.549010 13767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:20.549268 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 13767
I20260620 14:48:20.549399 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:48:20.549746 9079 external_mini_cluster.cc:1428] Setting key 6e6e5aaed3d7da3adfab2988a89f0107
I20260620 14:48:20.552204 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33137
--builtin_ntp_servers=127.8.221.212:44375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 14:48:20.557058 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:20.551202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:47063 (local address 127.8.221.254:33137)
0620 14:48:20.551331 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:20.551335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:20.552873 (+ 1538us) server_negotiation.cc:407] Connection header received
0620 14:48:20.555037 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:20.555040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:20.555081 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:20.555171 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:20.556528 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:20.556530 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:20.556532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:20.556543 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:20.556562 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:20.556569 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:20.556688 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:20.556910 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 14:48:20.556952 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:48:20.558532 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:20.551837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:33137 (local address 127.8.221.193:47063)
0620 14:48:20.552663 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:20.552737 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:20.554760 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:20.555394 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:20.555406 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:20.555528 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:20.556169 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:20.556173 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:20.556233 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:20.556362 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:20.556693 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:20.556705 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:20.556715 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 14:48:20.556931 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 14:48:20.557240 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":160,"threads_started":1}
I20260620 14:48:20.560099 13895 heartbeater.cc:347] Connected to a master server at 127.8.221.254:33137
I20260620 14:48:20.560395 13895 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:20.560993 13895 heartbeater.cc:511] Master 127.8.221.254:33137 requested a full tablet report, sending...
I20260620 14:48:20.563176 13705 ts_manager.cc:194] Registered new tserver with Master: 06dd2c627f8f4838ab6f2fda42e19aaf (127.8.221.193:40049)
W20260620 14:48:20.679147 13898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:20.679445 13898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:20.679489 13898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:20.683522 13898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:20.683573 13898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:20.683631 13898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:48:20.688374 13898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:20.689689 13898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:20.691367 13898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:20.702842 13904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:20.702877 13903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:20.703075 13898 server_base.cc:1034] running on GCE node
W20260620 14:48:20.702850 13906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:20.703699 13898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:20.704456 13898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:20.705657 13898 hybrid_clock.cc:648] HybridClock initialized: now 1781966900705623 us; error 52 us; skew 500 ppm
I20260620 14:48:20.707922 13898 webserver.cc:460] Webserver started at http://127.8.221.194:37011/ using document root <none> and password file <none>
I20260620 14:48:20.708607 13898 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:20.708673 13898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:20.708909 13898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:20.711431 13898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "bd0ced7dd2b34af1a2fb515f7b3c3871"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-65tz"
server_key: "0c98edca571a4e51a98114645e9f8d28"
server_key_iv: "accc8b2fc4278b50def6269489f3ad76"
server_key_version: "encryptionkey@0"
I20260620 14:48:20.712119 13898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "bd0ced7dd2b34af1a2fb515f7b3c3871"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-65tz"
server_key: "0c98edca571a4e51a98114645e9f8d28"
server_key_iv: "accc8b2fc4278b50def6269489f3ad76"
server_key_version: "encryptionkey@0"
I20260620 14:48:20.716325 13898 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:48:20.719262 13912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:20.720714 13898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:48:20.720847 13898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "bd0ced7dd2b34af1a2fb515f7b3c3871"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-65tz"
server_key: "0c98edca571a4e51a98114645e9f8d28"
server_key_iv: "accc8b2fc4278b50def6269489f3ad76"
server_key_version: "encryptionkey@0"
I20260620 14:48:20.720963 13898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:20.726089 13898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:20.726755 13898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:20.726946 13898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:20.727563 13898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:20.728591 13898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:20.728648 13898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:20.728725 13898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:20.728762 13898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:20.739995 13898 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:40335
I20260620 14:48:20.740025 14025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:40335 every 8 connection(s)
I20260620 14:48:20.741129 13898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:20.748107 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:20.743343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:42133 (local address 127.8.221.254:33137)
0620 14:48:20.743564 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:20.743571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:20.744858 (+ 1287us) server_negotiation.cc:407] Connection header received
0620 14:48:20.746352 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:20.746356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:20.746407 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:20.746521 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:20.747675 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:20.747679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:20.747681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:20.747691 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:20.747713 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:20.747721 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:20.747805 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:20.747930 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:48:20.747975 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 14:48:20.749097 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:20.743878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:33137 (local address 127.8.221.194:42133)
0620 14:48:20.744682 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:20.744728 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:20.746062 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:20.746697 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:20.746704 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:20.746798 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:20.747410 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:20.747411 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:20.747453 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:20.747522 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:20.747802 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:20.747809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:20.747813 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:48:20.747957 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 14:48:20.748180 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":127,"threads_started":1}
I20260620 14:48:20.750077 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 13898
I20260620 14:48:20.750202 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:20.750504 9079 external_mini_cluster.cc:1428] Setting key 26b2c7e07d30647b83ab3e4e74b5a702
I20260620 14:48:20.750531 14026 heartbeater.cc:347] Connected to a master server at 127.8.221.254:33137
I20260620 14:48:20.750778 14026 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:20.751282 14026 heartbeater.cc:511] Master 127.8.221.254:33137 requested a full tablet report, sending...
I20260620 14:48:20.752462 13705 ts_manager.cc:194] Registered new tserver with Master: bd0ced7dd2b34af1a2fb515f7b3c3871 (127.8.221.194:40335)
I20260620 14:48:20.752765 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33137
--builtin_ntp_servers=127.8.221.212:44375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:48:20.884466 14029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:20.884804 14029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:20.884852 14029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:20.889323 14029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:20.889400 14029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:20.889511 14029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:48:20.894686 14029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:20.896200 14029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:20.898206 14029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:20.909879 14034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:20.909858 14035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:20.909996 14029 server_base.cc:1034] running on GCE node
W20260620 14:48:20.909885 14037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:20.910708 14029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:20.911580 14029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:20.912889 14029 hybrid_clock.cc:648] HybridClock initialized: now 1781966900912855 us; error 46 us; skew 500 ppm
I20260620 14:48:20.915299 14029 webserver.cc:460] Webserver started at http://127.8.221.195:46239/ using document root <none> and password file <none>
I20260620 14:48:20.916035 14029 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:20.916144 14029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:20.916419 14029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:20.919299 14029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "20accd3424d645dd98f35692cb1a12f9"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-65tz"
server_key: "5673683464e42f2cd70bb9229e216232"
server_key_iv: "78748bc0997fa97b6db3a6c70583bd76"
server_key_version: "encryptionkey@0"
I20260620 14:48:20.920147 14029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "20accd3424d645dd98f35692cb1a12f9"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-65tz"
server_key: "5673683464e42f2cd70bb9229e216232"
server_key_iv: "78748bc0997fa97b6db3a6c70583bd76"
server_key_version: "encryptionkey@0"
I20260620 14:48:20.925071 14029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:48:20.928448 14043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:20.930147 14029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:20.930302 14029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "20accd3424d645dd98f35692cb1a12f9"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-65tz"
server_key: "5673683464e42f2cd70bb9229e216232"
server_key_iv: "78748bc0997fa97b6db3a6c70583bd76"
server_key_version: "encryptionkey@0"
I20260620 14:48:20.930433 14029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:20.936196 14029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:20.936945 14029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:20.937152 14029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:20.937851 14029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:20.938934 14029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:20.939006 14029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:20.939079 14029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:20.939133 14029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:20.951608 14029 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:39803
I20260620 14:48:20.951629 14156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:39803 every 8 connection(s)
I20260620 14:48:20.952685 14029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:20.960124 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:20.954968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:37819 (local address 127.8.221.254:33137)
0620 14:48:20.955175 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:20.955179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:20.956483 (+ 1304us) server_negotiation.cc:407] Connection header received
0620 14:48:20.958045 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:20.958048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:20.958120 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:20.958208 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:20.959569 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:20.959572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:20.959575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:20.959586 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:20.959610 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:20.959618 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:20.959707 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:20.959875 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 14:48:20.959934 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 14:48:20.960167 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 14029
I20260620 14:48:20.960286 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:20.960680 9079 external_mini_cluster.cc:1428] Setting key 7c59421e4ece0506fd219308b40b4818
I20260620 14:48:20.961026 14159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:20.955477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:33137 (local address 127.8.221.195:37819)
0620 14:48:20.956296 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:20.956347 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:20.957757 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:20.958412 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:20.958424 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:20.958529 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:20.959304 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:20.959305 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:20.959347 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:20.959416 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:20.959715 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:20.959721 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:20.959725 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:48:20.959891 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 14:48:20.960090 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":141,"threads_started":1}
I20260620 14:48:20.962406 14157 heartbeater.cc:347] Connected to a master server at 127.8.221.254:33137
I20260620 14:48:20.962750 14157 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:20.963302 14157 heartbeater.cc:511] Master 127.8.221.254:33137 requested a full tablet report, sending...
I20260620 14:48:20.964497 13705 ts_manager.cc:194] Registered new tserver with Master: 20accd3424d645dd98f35692cb1a12f9 (127.8.221.195:39803)
I20260620 14:48:20.975538 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:48:20.979871 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:20.977965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41306 (local address 127.8.221.254:33137)
0620 14:48:20.978185 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:20.978190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:20.978350 (+ 160us) server_negotiation.cc:407] Connection header received
0620 14:48:20.978515 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:20.978519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:20.978565 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:20.978656 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:20.979419 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:20.979422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:20.979425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:20.979436 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:20.979463 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:20.979471 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:20.979560 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:20.979640 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:48:20.979685 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 14:48:20.983253 13705 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:48:20.986737 13705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:21.000010 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:20.996661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:39803 (local address 127.0.0.1:53988)
0620 14:48:20.997392 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:20.997405 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:20.997468 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:20.998445 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:20.998448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:20.998470 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:20.999211 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:20.999213 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:20.999234 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:20.999311 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:20.999666 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:20.999675 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:20.999681 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:48:20.999772 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:48:20.999855 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":245,"threads_started":1}
I20260620 14:48:21.000484 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:20.997032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53988 (local address 127.8.221.195:39803)
0620 14:48:20.998031 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:20.998046 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:20.998069 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:48:20.998121 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:20.998128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:20.998290 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:20.998736 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:20.999439 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:20.999447 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:20.999462 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:20.999473 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:20.999517 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:20.999549 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:21.000228 (+ 679us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:21.000280 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 14:48:21.000352 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":892,"thread_start_us":189,"threads_started":1}
I20260620 14:48:21.000818 14172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:20.997291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:40335 (local address 127.0.0.1:33498)
0620 14:48:20.997629 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:20.997641 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:20.997706 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:20.999480 (+ 1774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:20.999483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:20.999498 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:20.999949 (+ 451us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:20.999951 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:20.999961 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:20.999984 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:21.000552 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:21.000557 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:21.000561 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:48:21.000654 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:48:21.000702 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":115,"threads_started":1}
I20260620 14:48:21.001297 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:20.996362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:40049 (local address 127.0.0.1:37126)
0620 14:48:20.997244 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:20.997292 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:20.997406 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:20.998225 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:20.998233 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:20.998356 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:20.999366 (+ 1010us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:20.999371 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:20.999388 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:20.999445 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:21.000820 (+ 1375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:21.000824 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:21.000828 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:48:21.001083 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 14:48:21.001131 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":79,"threads_started":1}
I20260620 14:48:21.001369 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:20.997760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33498 (local address 127.8.221.194:40335)
0620 14:48:20.998880 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:20.998896 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:20.998940 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:48:20.999009 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:20.999019 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:20.999276 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:20.999509 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:21.000319 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:21.000324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:21.000334 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:21.000343 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:21.000405 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:21.000432 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:21.000569 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:21.001093 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 14:48:21.001177 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":980,"thread_start_us":142,"threads_started":1}
I20260620 14:48:21.002240 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:20.997002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.8.221.193:40049)
0620 14:48:20.997658 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:20.997684 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:20.997709 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:48:20.997767 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:20.997775 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:20.997994 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:20.998186 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:20.999622 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:20.999630 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:20.999950 (+ 320us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:20.999976 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:21.000041 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:21.000474 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:21.001874 (+ 1400us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:21.001939 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 14:48:21.002035 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":98,"threads_started":1}
I20260620 14:48:21.003357 14091 tablet_service.cc:1467] Processing CreateTablet for tablet b284d6b9347d443b865b6996978a69cb (DEFAULT_TABLE table=test-table [id=5b4729362bb54a63b24c2c1b04d1c126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:21.004307 14091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b284d6b9347d443b865b6996978a69cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:21.005007 13829 tablet_service.cc:1467] Processing CreateTablet for tablet b284d6b9347d443b865b6996978a69cb (DEFAULT_TABLE table=test-table [id=5b4729362bb54a63b24c2c1b04d1c126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:21.005007 13960 tablet_service.cc:1467] Processing CreateTablet for tablet b284d6b9347d443b865b6996978a69cb (DEFAULT_TABLE table=test-table [id=5b4729362bb54a63b24c2c1b04d1c126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:21.006313 13960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b284d6b9347d443b865b6996978a69cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:21.006313 13829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b284d6b9347d443b865b6996978a69cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:21.011013 14174 tablet_bootstrap.cc:492] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9: Bootstrap starting.
I20260620 14:48:21.013051 14174 tablet_bootstrap.cc:654] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:21.013773 14174 log.cc:824] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:21.015525 14176 tablet_bootstrap.cc:492] T b284d6b9347d443b865b6996978a69cb P 06dd2c627f8f4838ab6f2fda42e19aaf: Bootstrap starting.
I20260620 14:48:21.015762 14174 tablet_bootstrap.cc:492] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9: No bootstrap required, opened a new log
I20260620 14:48:21.015929 14174 ts_tablet_manager.cc:1397] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:48:21.017241 14177 tablet_bootstrap.cc:492] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871: Bootstrap starting.
I20260620 14:48:21.017758 14176 tablet_bootstrap.cc:654] T b284d6b9347d443b865b6996978a69cb P 06dd2c627f8f4838ab6f2fda42e19aaf: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:21.018594 14176 log.cc:824] T b284d6b9347d443b865b6996978a69cb P 06dd2c627f8f4838ab6f2fda42e19aaf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:21.020411 14174 raft_consensus.cc:348] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dd2c627f8f4838ab6f2fda42e19aaf" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40049 } } peers { permanent_uuid: "20accd3424d645dd98f35692cb1a12f9" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39803 } } peers { permanent_uuid: "bd0ced7dd2b34af1a2fb515f7b3c3871" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 40335 } }
I20260620 14:48:21.020687 14174 raft_consensus.cc:374] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:21.020745 14174 raft_consensus.cc:729] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20accd3424d645dd98f35692cb1a12f9, State: Initialized, Role: FOLLOWER
I20260620 14:48:21.020771 14177 tablet_bootstrap.cc:654] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:21.020854 14176 tablet_bootstrap.cc:492] T b284d6b9347d443b865b6996978a69cb P 06dd2c627f8f4838ab6f2fda42e19aaf: No bootstrap required, opened a new log
I20260620 14:48:21.021016 14176 ts_tablet_manager.cc:1397] T b284d6b9347d443b865b6996978a69cb P 06dd2c627f8f4838ab6f2fda42e19aaf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:48:21.021235 14174 consensus_queue.cc:260] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dd2c627f8f4838ab6f2fda42e19aaf" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40049 } } peers { permanent_uuid: "20accd3424d645dd98f35692cb1a12f9" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39803 } } peers { permanent_uuid: "bd0ced7dd2b34af1a2fb515f7b3c3871" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 40335 } }
I20260620 14:48:21.021957 14177 log.cc:824] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:21.022104 14157 heartbeater.cc:503] Master 127.8.221.254:33137 was elected leader, sending a full tablet report...
I20260620 14:48:21.022264 14174 ts_tablet_manager.cc:1428] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:48:21.024364 14177 tablet_bootstrap.cc:492] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871: No bootstrap required, opened a new log
I20260620 14:48:21.024600 14177 ts_tablet_manager.cc:1397] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 14:48:21.025279 14176 raft_consensus.cc:348] T b284d6b9347d443b865b6996978a69cb P 06dd2c627f8f4838ab6f2fda42e19aaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dd2c627f8f4838ab6f2fda42e19aaf" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40049 } } peers { permanent_uuid: "20accd3424d645dd98f35692cb1a12f9" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39803 } } peers { permanent_uuid: "bd0ced7dd2b34af1a2fb515f7b3c3871" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 40335 } }
I20260620 14:48:21.025533 14176 raft_consensus.cc:374] T b284d6b9347d443b865b6996978a69cb P 06dd2c627f8f4838ab6f2fda42e19aaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:21.025581 14176 raft_consensus.cc:729] T b284d6b9347d443b865b6996978a69cb P 06dd2c627f8f4838ab6f2fda42e19aaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06dd2c627f8f4838ab6f2fda42e19aaf, State: Initialized, Role: FOLLOWER
I20260620 14:48:21.026042 14176 consensus_queue.cc:260] T b284d6b9347d443b865b6996978a69cb P 06dd2c627f8f4838ab6f2fda42e19aaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dd2c627f8f4838ab6f2fda42e19aaf" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40049 } } peers { permanent_uuid: "20accd3424d645dd98f35692cb1a12f9" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39803 } } peers { permanent_uuid: "bd0ced7dd2b34af1a2fb515f7b3c3871" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 40335 } }
I20260620 14:48:21.026962 14176 ts_tablet_manager.cc:1428] T b284d6b9347d443b865b6996978a69cb P 06dd2c627f8f4838ab6f2fda42e19aaf: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:48:21.027522 13895 heartbeater.cc:503] Master 127.8.221.254:33137 was elected leader, sending a full tablet report...
I20260620 14:48:21.029170 14177 raft_consensus.cc:348] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dd2c627f8f4838ab6f2fda42e19aaf" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40049 } } peers { permanent_uuid: "20accd3424d645dd98f35692cb1a12f9" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39803 } } peers { permanent_uuid: "bd0ced7dd2b34af1a2fb515f7b3c3871" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 40335 } }
I20260620 14:48:21.029448 14177 raft_consensus.cc:374] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:21.029502 14177 raft_consensus.cc:729] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd0ced7dd2b34af1a2fb515f7b3c3871, State: Initialized, Role: FOLLOWER
I20260620 14:48:21.029924 14177 consensus_queue.cc:260] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dd2c627f8f4838ab6f2fda42e19aaf" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40049 } } peers { permanent_uuid: "20accd3424d645dd98f35692cb1a12f9" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39803 } } peers { permanent_uuid: "bd0ced7dd2b34af1a2fb515f7b3c3871" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 40335 } }
I20260620 14:48:21.030646 14177 ts_tablet_manager.cc:1428] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:48:21.030730 14026 heartbeater.cc:503] Master 127.8.221.254:33137 was elected leader, sending a full tablet report...
W20260620 14:48:21.050321 13896 tablet.cc:2369] T b284d6b9347d443b865b6996978a69cb P 06dd2c627f8f4838ab6f2fda42e19aaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:48:21.203799 14158 tablet.cc:2369] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:48:21.242548 14027 tablet.cc:2369] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:48:21.386165 14182 raft_consensus.cc:482] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:21.386494 14182 raft_consensus.cc:504] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dd2c627f8f4838ab6f2fda42e19aaf" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40049 } } peers { permanent_uuid: "20accd3424d645dd98f35692cb1a12f9" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39803 } } peers { permanent_uuid: "bd0ced7dd2b34af1a2fb515f7b3c3871" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 40335 } }
I20260620 14:48:21.387748 14182 leader_election.cc:288] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06dd2c627f8f4838ab6f2fda42e19aaf (127.8.221.193:40049), 20accd3424d645dd98f35692cb1a12f9 (127.8.221.195:39803)
I20260620 14:48:21.390784 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:21.388337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:36729 (local address 127.8.221.195:39803)
0620 14:48:21.388544 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:21.388552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:21.388928 (+ 376us) server_negotiation.cc:407] Connection header received
0620 14:48:21.388995 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:21.388999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:21.389057 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:21.389173 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:21.390297 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:21.390301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:21.390305 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:21.390317 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:21.390342 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:21.390354 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:21.390462 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:21.390571 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 14:48:21.390620 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 14:48:21.390784 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:21.388321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:48161 (local address 127.8.221.193:40049)
0620 14:48:21.388480 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:21.388486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:21.388837 (+ 351us) server_negotiation.cc:407] Connection header received
0620 14:48:21.388996 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:21.389000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:21.389057 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:21.389173 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:21.390297 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:21.390301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:21.390304 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:21.390317 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:21.390343 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:21.390363 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:21.390462 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:21.390571 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 14:48:21.390620 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:48:21.390820 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:21.388171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:39803 (local address 127.8.221.194:36729)
0620 14:48:21.388698 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:21.388715 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:21.388841 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:21.389217 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:21.389220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:21.389237 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:21.390039 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:21.390040 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:21.390072 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:21.390107 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:21.390463 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:21.390465 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:21.390468 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:48:21.390560 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 14:48:21.390622 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":80,"spinlock_wait_cycles":9472,"thread_start_us":128,"threads_started":1}
I20260620 14:48:21.390820 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:21.388171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:40049 (local address 127.8.221.194:48161)
0620 14:48:21.388665 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:21.388683 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:21.388841 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:21.389217 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:21.389220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:21.389237 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:21.390105 (+ 868us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:21.390107 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:21.390115 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:21.390133 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:21.390459 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:21.390463 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:21.390466 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:48:21.390560 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 14:48:21.390615 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"spinlock_wait_cycles":29056,"thread_start_us":166,"threads_started":1}
I20260620 14:48:21.391589 14111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b284d6b9347d443b865b6996978a69cb" candidate_uuid: "bd0ced7dd2b34af1a2fb515f7b3c3871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20accd3424d645dd98f35692cb1a12f9" is_pre_election: true
I20260620 14:48:21.391945 14111 raft_consensus.cc:2454] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd0ced7dd2b34af1a2fb515f7b3c3871 in term 0.
I20260620 14:48:21.392217 13849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b284d6b9347d443b865b6996978a69cb" candidate_uuid: "bd0ced7dd2b34af1a2fb515f7b3c3871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06dd2c627f8f4838ab6f2fda42e19aaf" is_pre_election: true
I20260620 14:48:21.392390 13915 leader_election.cc:302] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20accd3424d645dd98f35692cb1a12f9, bd0ced7dd2b34af1a2fb515f7b3c3871; no voters:
I20260620 14:48:21.392632 14182 raft_consensus.cc:2781] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:21.392647 13849 raft_consensus.cc:2454] T b284d6b9347d443b865b6996978a69cb P 06dd2c627f8f4838ab6f2fda42e19aaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd0ced7dd2b34af1a2fb515f7b3c3871 in term 0.
I20260620 14:48:21.392719 14182 raft_consensus.cc:482] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:21.392755 14182 raft_consensus.cc:3037] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:21.393971 14182 raft_consensus.cc:504] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dd2c627f8f4838ab6f2fda42e19aaf" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40049 } } peers { permanent_uuid: "20accd3424d645dd98f35692cb1a12f9" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39803 } } peers { permanent_uuid: "bd0ced7dd2b34af1a2fb515f7b3c3871" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 40335 } }
I20260620 14:48:21.394317 14182 leader_election.cc:288] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [CANDIDATE]: Term 1 election: Requested vote from peers 06dd2c627f8f4838ab6f2fda42e19aaf (127.8.221.193:40049), 20accd3424d645dd98f35692cb1a12f9 (127.8.221.195:39803)
I20260620 14:48:21.394670 13849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b284d6b9347d443b865b6996978a69cb" candidate_uuid: "bd0ced7dd2b34af1a2fb515f7b3c3871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06dd2c627f8f4838ab6f2fda42e19aaf"
I20260620 14:48:21.394785 14111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b284d6b9347d443b865b6996978a69cb" candidate_uuid: "bd0ced7dd2b34af1a2fb515f7b3c3871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20accd3424d645dd98f35692cb1a12f9"
I20260620 14:48:21.394816 13849 raft_consensus.cc:3037] T b284d6b9347d443b865b6996978a69cb P 06dd2c627f8f4838ab6f2fda42e19aaf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:21.394923 14111 raft_consensus.cc:3037] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:21.396044 14111 raft_consensus.cc:2454] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd0ced7dd2b34af1a2fb515f7b3c3871 in term 1.
I20260620 14:48:21.396359 13915 leader_election.cc:302] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20accd3424d645dd98f35692cb1a12f9, bd0ced7dd2b34af1a2fb515f7b3c3871; no voters:
I20260620 14:48:21.396431 13849 raft_consensus.cc:2454] T b284d6b9347d443b865b6996978a69cb P 06dd2c627f8f4838ab6f2fda42e19aaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd0ced7dd2b34af1a2fb515f7b3c3871 in term 1.
I20260620 14:48:21.396551 14182 raft_consensus.cc:2781] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:21.396771 14182 raft_consensus.cc:686] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [term 1 LEADER]: Becoming Leader. State: Replica: bd0ced7dd2b34af1a2fb515f7b3c3871, State: Running, Role: LEADER
I20260620 14:48:21.397162 14182 consensus_queue.cc:237] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dd2c627f8f4838ab6f2fda42e19aaf" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40049 } } peers { permanent_uuid: "20accd3424d645dd98f35692cb1a12f9" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39803 } } peers { permanent_uuid: "bd0ced7dd2b34af1a2fb515f7b3c3871" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 40335 } }
I20260620 14:48:21.401197 13706 catalog_manager.cc:5495] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd0ced7dd2b34af1a2fb515f7b3c3871 (127.8.221.194). New cstate: current_term: 1 leader_uuid: "bd0ced7dd2b34af1a2fb515f7b3c3871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06dd2c627f8f4838ab6f2fda42e19aaf" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20accd3424d645dd98f35692cb1a12f9" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd0ced7dd2b34af1a2fb515f7b3c3871" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 40335 } health_report { overall_health: HEALTHY } } }
I20260620 14:48:21.464072 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:21.462163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33514 (local address 127.8.221.194:40335)
0620 14:48:21.462322 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:21.462326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:21.462343 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:48:21.462552 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:21.462555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:21.462651 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:21.462740 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:21.463647 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:21.463650 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:21.463652 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:21.463661 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:21.463678 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:21.463686 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:21.463765 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:21.463859 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 14:48:21.463892 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:48:21.471149 14111 raft_consensus.cc:1261] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9 [term 1 FOLLOWER]: Refusing update from remote peer bd0ced7dd2b34af1a2fb515f7b3c3871: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:21.471149 13849 raft_consensus.cc:1261] T b284d6b9347d443b865b6996978a69cb P 06dd2c627f8f4838ab6f2fda42e19aaf [term 1 FOLLOWER]: Refusing update from remote peer bd0ced7dd2b34af1a2fb515f7b3c3871: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:21.472164 14185 consensus_queue.cc:1035] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20accd3424d645dd98f35692cb1a12f9" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:21.472388 14182 consensus_queue.cc:1035] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06dd2c627f8f4838ab6f2fda42e19aaf" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:21.493462 13706 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:21.493654 13706 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:21.496336 13706 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40ec534bdca5490b818af016775ece0a: Sending DeleteTablet for 3 replicas of tablet b284d6b9347d443b865b6996978a69cb
I20260620 14:48:21.497155 14091 tablet_service.cc:1514] Processing DeleteTablet for tablet b284d6b9347d443b865b6996978a69cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:21 UTC) from {username='slave'} at 127.0.0.1:53988
I20260620 14:48:21.497418 13829 tablet_service.cc:1514] Processing DeleteTablet for tablet b284d6b9347d443b865b6996978a69cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:21 UTC) from {username='slave'} at 127.0.0.1:37126
I20260620 14:48:21.497766 14198 tablet_replica.cc:330] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9: stopping tablet replica
I20260620 14:48:21.498049 13960 tablet_service.cc:1514] Processing DeleteTablet for tablet b284d6b9347d443b865b6996978a69cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:21 UTC) from {username='slave'} at 127.0.0.1:33498
I20260620 14:48:21.498140 14198 raft_consensus.cc:2229] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:21.498260 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 13767
I20260620 14:48:21.498587 14198 raft_consensus.cc:2258] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:21.498952 14200 tablet_replica.cc:330] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871: stopping tablet replica
I20260620 14:48:21.499315 14200 raft_consensus.cc:2229] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:21.499806 14200 raft_consensus.cc:2258] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:21.501601 14200 ts_tablet_manager.cc:1905] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:21.501844 14198 ts_tablet_manager.cc:1905] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:21.505115 14200 ts_tablet_manager.cc:1918] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:21.505231 14200 log.cc:1196] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/b284d6b9347d443b865b6996978a69cb
I20260620 14:48:21.505267 14198 ts_tablet_manager.cc:1918] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:21.505477 14198 log.cc:1196] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/b284d6b9347d443b865b6996978a69cb
I20260620 14:48:21.505615 14200 ts_tablet_manager.cc:1939] T b284d6b9347d443b865b6996978a69cb P bd0ced7dd2b34af1a2fb515f7b3c3871: Deleting consensus metadata
I20260620 14:48:21.505946 14198 ts_tablet_manager.cc:1939] T b284d6b9347d443b865b6996978a69cb P 20accd3424d645dd98f35692cb1a12f9: Deleting consensus metadata
I20260620 14:48:21.507166 13693 catalog_manager.cc:4841] TS bd0ced7dd2b34af1a2fb515f7b3c3871 (127.8.221.194:40335): tablet b284d6b9347d443b865b6996978a69cb (table test-table [id=5b4729362bb54a63b24c2c1b04d1c126]) successfully deleted
I20260620 14:48:21.507427 13693 catalog_manager.cc:4841] TS 20accd3424d645dd98f35692cb1a12f9 (127.8.221.195:39803): tablet b284d6b9347d443b865b6996978a69cb (table test-table [id=5b4729362bb54a63b24c2c1b04d1c126]) successfully deleted
W20260620 14:48:21.508585 13691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.221.193:40049 (error 108)
I20260620 14:48:21.509387 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 13898
I20260620 14:48:21.514091 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:21.509363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:40049 (local address 127.0.0.1:37130)
0620 14:48:21.513942 (+ 4579us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:21.514010 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.193:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4491,"thread_start_us":71,"threads_started":1}
W20260620 14:48:21.514286 13691 catalog_manager.cc:4584] TS 06dd2c627f8f4838ab6f2fda42e19aaf (127.8.221.193:40049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b284d6b9347d443b865b6996978a69cb: Network error: Client connection negotiation failed: client connection to 127.8.221.193:40049: connect: Connection refused (error 111)
I20260620 14:48:21.516328 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 14029
I20260620 14:48:21.522444 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 13676
2026-06-20T14:48:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1472 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 14:48:21.544874 9079 mini_oidc.cc:152] Starting JWKS server
I20260620 14:48:21.546613 9079 webserver.cc:460] Webserver started at https://127.0.0.1:40129/ using document root <none> and password file <none>
I20260620 14:48:21.546803 9079 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:48:21.547070 9079 webserver.cc:460] Webserver started at http://127.0.0.1:46019/ using document root <none> and password file <none>
2026-06-20T14:48:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:21Z Disabled control of system clock
I20260620 14:48:21.556882 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:39127
--webserver_interface=127.8.221.254
--webserver_port=0
--builtin_ntp_servers=127.8.221.212:44375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:39127
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40129/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0/testchainca.cert with env {}
W20260620 14:48:21.671051 14208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:21.671334 14208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:21.671373 14208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:21.675567 14208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:21.675629 14208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:21.675645 14208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:21.675662 14208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:21.675678 14208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:48:21.675695 14208 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:48:21.675709 14208 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40129/jwks.json
I20260620 14:48:21.681098 14208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:39127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:39127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40129/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:21.682313 14208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:21.683355 14208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:21.692246 14216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:21.692267 14213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:21.692469 14214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:21.692694 14208 server_base.cc:1034] running on GCE node
I20260620 14:48:21.693245 14208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:21.694437 14208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:21.695659 14208 hybrid_clock.cc:648] HybridClock initialized: now 1781966901695638 us; error 38 us; skew 500 ppm
I20260620 14:48:21.697640 14208 webserver.cc:460] Webserver started at http://127.8.221.254:38689/ using document root <none> and password file <none>
I20260620 14:48:21.698305 14208 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:21.698403 14208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:21.698642 14208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:21.700690 14208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "cfa29c3eae2b40498c2bfc7140fdad26"
format_stamp: "Formatted at 2026-06-20 14:48:21 on dist-test-slave-65tz"
server_key: "c226623e6fe90644ab628e91e147e1f4"
server_key_iv: "21bb5e99ceb51b584bef0ae1cb7a6d89"
server_key_version: "encryptionkey@0"
I20260620 14:48:21.701261 14208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "cfa29c3eae2b40498c2bfc7140fdad26"
format_stamp: "Formatted at 2026-06-20 14:48:21 on dist-test-slave-65tz"
server_key: "c226623e6fe90644ab628e91e147e1f4"
server_key_iv: "21bb5e99ceb51b584bef0ae1cb7a6d89"
server_key_version: "encryptionkey@0"
I20260620 14:48:21.705277 14208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:48:21.708089 14222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:21.709748 14208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:48:21.709919 14208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "cfa29c3eae2b40498c2bfc7140fdad26"
format_stamp: "Formatted at 2026-06-20 14:48:21 on dist-test-slave-65tz"
server_key: "c226623e6fe90644ab628e91e147e1f4"
server_key_iv: "21bb5e99ceb51b584bef0ae1cb7a6d89"
server_key_version: "encryptionkey@0"
I20260620 14:48:21.710032 14208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:21.737269 14208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:21.738106 14208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:21.738315 14208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:21.745841 14208 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:39127
I20260620 14:48:21.745951 14274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:39127 every 8 connection(s)
I20260620 14:48:21.746833 14208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:21.751112 14275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:21.753733 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 14208
I20260620 14:48:21.753930 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:48:21.754254 9079 external_mini_cluster.cc:1428] Setting key e80c481445c32c6e8148a4bbcb6dcbde
I20260620 14:48:21.758708 14275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26: Bootstrap starting.
I20260620 14:48:21.761891 14275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:21.762988 14275 log.cc:824] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:21.765045 14278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:21.755835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56334 (local address 127.8.221.254:39127)
0620 14:48:21.756491 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:21.756506 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:21.756543 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:48:21.757558 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:21.757577 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:21.758088 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:21.758390 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:21.758778 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:21.759815 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:21.761190 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:21.761705 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:21.761933 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:21.762877 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:21.762882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:21.762896 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:21.763364 (+ 468us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:21.763450 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:21.763482 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:21.763593 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:21.763795 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 14:48:21.764097 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":132,"threads_started":1}
I20260620 14:48:21.765581 14275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26: No bootstrap required, opened a new log
I20260620 14:48:21.768757 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:48:21.769918 14275 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa29c3eae2b40498c2bfc7140fdad26" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 39127 } }
I20260620 14:48:21.770275 14275 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:21.770359 14275 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfa29c3eae2b40498c2bfc7140fdad26, State: Initialized, Role: FOLLOWER
I20260620 14:48:21.770828 14275 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa29c3eae2b40498c2bfc7140fdad26" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 39127 } }
I20260620 14:48:21.771010 14275 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:21.771091 14275 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:21.771201 14275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:21.772231 14275 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa29c3eae2b40498c2bfc7140fdad26" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 39127 } }
I20260620 14:48:21.772586 14275 leader_election.cc:302] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfa29c3eae2b40498c2bfc7140fdad26; no voters:
I20260620 14:48:21.772957 14275 leader_election.cc:288] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:21.773150 14280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:21.773792 14280 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26 [term 1 LEADER]: Becoming Leader. State: Replica: cfa29c3eae2b40498c2bfc7140fdad26, State: Running, Role: LEADER
I20260620 14:48:21.774281 14280 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa29c3eae2b40498c2bfc7140fdad26" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 39127 } }
I20260620 14:48:21.774402 14275 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:21.776410 14282 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfa29c3eae2b40498c2bfc7140fdad26. Latest consensus state: current_term: 1 leader_uuid: "cfa29c3eae2b40498c2bfc7140fdad26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa29c3eae2b40498c2bfc7140fdad26" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 39127 } } }
I20260620 14:48:21.776463 14281 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfa29c3eae2b40498c2bfc7140fdad26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfa29c3eae2b40498c2bfc7140fdad26" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 39127 } } }
I20260620 14:48:21.776547 14282 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:21.776573 14281 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26 [sys.catalog]: This master's current role is: LEADER
W20260620 14:48:21.779590 14294 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:48:21.779696 14294 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:48:21.779807 14295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:21.780787 14295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:21.786247 14295 catalog_manager.cc:1347] Generated new cluster ID: 9ce14407b39f4b72a1fb2db3e561a604
I20260620 14:48:21.786320 14295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:21.800966 14295 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:21.802544 14295 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:21.811523 14295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfa29c3eae2b40498c2bfc7140fdad26: Generated new TSK 0
I20260620 14:48:21.812584 14295 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:48:21.848997 14305 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 14:48:21.849161 14305 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:48:21.849951 14278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:21.823602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.8.221.254:39127)
0620 14:48:21.823732 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:21.823735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:21.824054 (+ 319us) server_negotiation.cc:407] Connection header received
0620 14:48:21.824283 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:21.824286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:21.824343 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:21.824445 (+ 102us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:48:21.824694 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:21.825260 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:21.826200 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:21.826441 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:21.826552 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:21.826884 (+ 332us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:48:21.849409 (+ 22525us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:48:21.849716 (+ 307us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56342: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 14:48:21.850045 14278 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56342: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 14:48:21.851577 14304 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:21.823563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:39127 (local address 127.0.0.1:56342)
0620 14:48:21.823959 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:21.823973 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:21.824127 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:21.824459 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:21.824462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:21.824469 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:48:21.824588 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:21.824594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:21.825369 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:21.825374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:21.826079 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:21.826087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:21.826558 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:21.826562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:21.826654 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:21.826702 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:48:21.849745 (+ 23043us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:48:21.850124 (+ 379us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.221.254:39127: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":164,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.8.221.254:39127: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40129/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 14:48:21.852679 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 14208
2026-06-20T14:48:21Z chronyd exiting
I20260620 14:48:21.891060 9079 test_util.cc:175] -----------------------------------------------
I20260620 14:48:21.891211 9079 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966871461349-9079-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (347 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 14:48:21.892445 9079 mini_oidc.cc:152] Starting JWKS server
I20260620 14:48:21.893460 9079 webserver.cc:460] Webserver started at https://127.0.0.1:37317/ using document root <none> and password file <none>
I20260620 14:48:21.893600 9079 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:48:21.893934 9079 webserver.cc:460] Webserver started at http://127.0.0.1:42221/ using document root <none> and password file <none>
2026-06-20T14:48:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:21Z Disabled control of system clock
I20260620 14:48:21.904747 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:36841
--webserver_interface=127.8.221.254
--webserver_port=0
--builtin_ntp_servers=127.8.221.212:45701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:36841
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37317/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966871461349-9079-0/testchainca.cert with env {}
W20260620 14:48:22.040539 14311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:22.040875 14311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:22.040923 14311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:22.045744 14311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:22.045859 14311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:22.045878 14311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:22.045898 14311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:22.045917 14311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:48:22.045935 14311 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:48:22.045951 14311 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37317/jwks.json
I20260620 14:48:22.052448 14311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:45701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:36841
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:36841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37317/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966871461349-9079-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:22.053900 14311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:22.055169 14311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:22.066057 14317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:22.066150 14316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:22.066061 14319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:22.066345 14311 server_base.cc:1034] running on GCE node
I20260620 14:48:22.067229 14311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:22.068538 14311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:22.069721 14311 hybrid_clock.cc:648] HybridClock initialized: now 1781966902069697 us; error 39 us; skew 500 ppm
I20260620 14:48:22.071877 14311 webserver.cc:460] Webserver started at http://127.8.221.254:38039/ using document root <none> and password file <none>
I20260620 14:48:22.072567 14311 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:22.072638 14311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:22.072885 14311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:22.075320 14311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "214b430de2854019923a710d182dde7e"
format_stamp: "Formatted at 2026-06-20 14:48:22 on dist-test-slave-65tz"
server_key: "c3eb0688cdce226827cc36eb6982f74b"
server_key_iv: "6ceec3a048fe698a2df35c417e25333e"
server_key_version: "encryptionkey@0"
I20260620 14:48:22.075977 14311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "214b430de2854019923a710d182dde7e"
format_stamp: "Formatted at 2026-06-20 14:48:22 on dist-test-slave-65tz"
server_key: "c3eb0688cdce226827cc36eb6982f74b"
server_key_iv: "6ceec3a048fe698a2df35c417e25333e"
server_key_version: "encryptionkey@0"
I20260620 14:48:22.080440 14311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:48:22.083647 14325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:22.085266 14311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:22.085458 14311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "214b430de2854019923a710d182dde7e"
format_stamp: "Formatted at 2026-06-20 14:48:22 on dist-test-slave-65tz"
server_key: "c3eb0688cdce226827cc36eb6982f74b"
server_key_iv: "6ceec3a048fe698a2df35c417e25333e"
server_key_version: "encryptionkey@0"
I20260620 14:48:22.085638 14311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:22.098224 14311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:22.099068 14311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:22.099288 14311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:22.107831 14311 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:36841
I20260620 14:48:22.107843 14377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:36841 every 8 connection(s)
I20260620 14:48:22.109009 14311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:22.112561 14378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:22.112564 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 14311
I20260620 14:48:22.112701 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:48:22.113034 9079 external_mini_cluster.cc:1428] Setting key e9c12ca2e7e408420de61cc143a8dd61
I20260620 14:48:22.118840 14378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e: Bootstrap starting.
I20260620 14:48:22.121493 14378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:22.122280 14378 log.cc:824] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:22.124382 14381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:22.114733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57276 (local address 127.8.221.254:36841)
0620 14:48:22.115490 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:22.115514 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:22.115560 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:48:22.116414 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:22.116435 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:22.116932 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:22.117290 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:22.117720 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.118840 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.120139 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.120857 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.121025 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:22.122327 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:22.122334 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:22.122352 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:22.122854 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:22.122912 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:22.122932 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:22.123013 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:22.123224 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:48:22.123500 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":173,"threads_started":1}
I20260620 14:48:22.124603 14378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e: No bootstrap required, opened a new log
I20260620 14:48:22.127605 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:48:22.128515 14378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214b430de2854019923a710d182dde7e" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 36841 } }
I20260620 14:48:22.128774 14378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:22.128839 14378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214b430de2854019923a710d182dde7e, State: Initialized, Role: FOLLOWER
I20260620 14:48:22.129261 14378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214b430de2854019923a710d182dde7e" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 36841 } }
I20260620 14:48:22.129375 14378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:22.129441 14378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:22.129555 14378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:22.130458 14378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214b430de2854019923a710d182dde7e" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 36841 } }
I20260620 14:48:22.130739 14378 leader_election.cc:302] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 214b430de2854019923a710d182dde7e; no voters:
I20260620 14:48:22.130983 14378 leader_election.cc:288] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:22.131167 14383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:22.131516 14383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e [term 1 LEADER]: Becoming Leader. State: Replica: 214b430de2854019923a710d182dde7e, State: Running, Role: LEADER
I20260620 14:48:22.131888 14383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214b430de2854019923a710d182dde7e" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 36841 } }
I20260620 14:48:22.132234 14378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:22.133704 14384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "214b430de2854019923a710d182dde7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214b430de2854019923a710d182dde7e" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 36841 } } }
I20260620 14:48:22.133880 14384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:22.134225 14385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 214b430de2854019923a710d182dde7e. Latest consensus state: current_term: 1 leader_uuid: "214b430de2854019923a710d182dde7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214b430de2854019923a710d182dde7e" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 36841 } } }
I20260620 14:48:22.134335 14385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e [sys.catalog]: This master's current role is: LEADER
W20260620 14:48:22.137575 14398 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:48:22.137641 14398 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:48:22.137727 14394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:22.138449 14394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:22.145519 14394 catalog_manager.cc:1347] Generated new cluster ID: c52b97fbcd0e4001862be26e77ae9ed1
I20260620 14:48:22.145665 14394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:22.169098 14394 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:22.170202 14394 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:22.181187 14394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 214b430de2854019923a710d182dde7e: Generated new TSK 0
I20260620 14:48:22.181980 14394 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:48:22.211441 14408 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 14:48:22.211544 14408 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:48:22.212234 14381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:22.191508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57292 (local address 127.8.221.254:36841)
0620 14:48:22.191669 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:22.191673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:22.191852 (+ 179us) server_negotiation.cc:407] Connection header received
0620 14:48:22.192093 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:22.192096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:22.192148 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:22.192260 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:48:22.192571 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.193135 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.194604 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.194895 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.195005 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:22.195347 (+ 342us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:48:22.211873 (+ 16526us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:48:22.212108 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57292: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 14:48:22.212316 14381 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57292: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 14:48:22.212671 14407 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:22.191354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:36841 (local address 127.0.0.1:57292)
0620 14:48:22.191730 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:22.191749 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:22.191908 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:22.192289 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:22.192293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:22.192302 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:48:22.192446 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:22.192454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.193258 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.193261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:22.194433 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:22.194444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.195046 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.195050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:22.195142 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:22.195177 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:48:22.212144 (+ 16967us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:48:22.212434 (+ 290us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.221.254:36841: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":117,"threads_started":1}
I20260620 14:48:22.213581 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 14311
2026-06-20T14:48:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (362 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 14:48:22.255052 9079 mini_oidc.cc:152] Starting JWKS server
I20260620 14:48:22.256029 9079 webserver.cc:460] Webserver started at https://127.0.0.1:38901/ using document root <none> and password file <none>
I20260620 14:48:22.256119 9079 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:48:22.256330 9079 webserver.cc:460] Webserver started at http://127.0.0.1:32785/ using document root <none> and password file <none>
2026-06-20T14:48:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:22Z Disabled control of system clock
I20260620 14:48:22.267174 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:32857
--webserver_interface=127.8.221.254
--webserver_port=0
--builtin_ntp_servers=127.8.221.212:46391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:32857
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38901/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:48:22.392318 14414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:22.392642 14414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:22.392686 14414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:22.396752 14414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:22.396817 14414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:22.396832 14414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:22.396849 14414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:22.396865 14414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:48:22.396883 14414 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:48:22.396896 14414 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38901/jwks.json
I20260620 14:48:22.402032 14414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:46391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:32857
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:32857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38901/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:22.403793 14414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:22.406162 14414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:22.418385 14419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:22.418743 14422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:22.418887 14420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:22.418927 14414 server_base.cc:1034] running on GCE node
I20260620 14:48:22.419631 14414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:22.421226 14414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:22.422398 14414 hybrid_clock.cc:648] HybridClock initialized: now 1781966902422376 us; error 40 us; skew 500 ppm
I20260620 14:48:22.424785 14414 webserver.cc:460] Webserver started at http://127.8.221.254:41435/ using document root <none> and password file <none>
I20260620 14:48:22.425514 14414 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:22.425582 14414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:22.425890 14414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:22.428361 14414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "5196d2ebe5d34eff970c28b43a64a941"
format_stamp: "Formatted at 2026-06-20 14:48:22 on dist-test-slave-65tz"
server_key: "a2696bb16d263ec6e7ca53ad2173aead"
server_key_iv: "41c8961faf4f24a6d17d70f01548c961"
server_key_version: "encryptionkey@0"
I20260620 14:48:22.429004 14414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "5196d2ebe5d34eff970c28b43a64a941"
format_stamp: "Formatted at 2026-06-20 14:48:22 on dist-test-slave-65tz"
server_key: "a2696bb16d263ec6e7ca53ad2173aead"
server_key_iv: "41c8961faf4f24a6d17d70f01548c961"
server_key_version: "encryptionkey@0"
I20260620 14:48:22.433485 14414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 14:48:22.436444 14428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:22.438097 14414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 14:48:22.438246 14414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "5196d2ebe5d34eff970c28b43a64a941"
format_stamp: "Formatted at 2026-06-20 14:48:22 on dist-test-slave-65tz"
server_key: "a2696bb16d263ec6e7ca53ad2173aead"
server_key_iv: "41c8961faf4f24a6d17d70f01548c961"
server_key_version: "encryptionkey@0"
I20260620 14:48:22.438374 14414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:22.450186 14414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:22.450970 14414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:22.451179 14414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:22.459969 14414 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:32857
I20260620 14:48:22.460018 14480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:32857 every 8 connection(s)
I20260620 14:48:22.461171 14414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:22.464320 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 14414
I20260620 14:48:22.464457 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:48:22.464818 9079 external_mini_cluster.cc:1428] Setting key 8843419b470c14eccde079870b598487
I20260620 14:48:22.465016 14481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:22.472110 14481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941: Bootstrap starting.
I20260620 14:48:22.474679 14481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:22.475463 14484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:22.466424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45218 (local address 127.8.221.254:32857)
0620 14:48:22.467260 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:22.467279 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:22.467325 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:48:22.468093 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:22.468115 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:22.468593 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:22.468988 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:22.469494 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.470669 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.471590 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.472122 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.472306 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:22.473322 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:22.473328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:22.473343 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:22.473876 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:22.473940 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:22.473968 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:22.474080 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:22.474196 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 14:48:22.474521 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":168,"threads_started":1}
I20260620 14:48:22.475481 14481 log.cc:824] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:22.477749 14481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941: No bootstrap required, opened a new log
I20260620 14:48:22.478758 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:48:22.481355 14481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5196d2ebe5d34eff970c28b43a64a941" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 32857 } }
I20260620 14:48:22.481621 14481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:22.481709 14481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5196d2ebe5d34eff970c28b43a64a941, State: Initialized, Role: FOLLOWER
I20260620 14:48:22.482194 14481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5196d2ebe5d34eff970c28b43a64a941" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 32857 } }
I20260620 14:48:22.482352 14481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:22.482446 14481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:22.482576 14481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:22.483649 14481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5196d2ebe5d34eff970c28b43a64a941" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 32857 } }
I20260620 14:48:22.484009 14481 leader_election.cc:302] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5196d2ebe5d34eff970c28b43a64a941; no voters:
I20260620 14:48:22.484449 14481 leader_election.cc:288] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:22.484611 14486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:22.484905 14486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941 [term 1 LEADER]: Becoming Leader. State: Replica: 5196d2ebe5d34eff970c28b43a64a941, State: Running, Role: LEADER
I20260620 14:48:22.485289 14486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5196d2ebe5d34eff970c28b43a64a941" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 32857 } }
I20260620 14:48:22.485944 14481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:22.487602 14487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5196d2ebe5d34eff970c28b43a64a941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5196d2ebe5d34eff970c28b43a64a941" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 32857 } } }
I20260620 14:48:22.487774 14487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:22.487581 14488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5196d2ebe5d34eff970c28b43a64a941. Latest consensus state: current_term: 1 leader_uuid: "5196d2ebe5d34eff970c28b43a64a941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5196d2ebe5d34eff970c28b43a64a941" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 32857 } } }
I20260620 14:48:22.488090 14488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941 [sys.catalog]: This master's current role is: LEADER
W20260620 14:48:22.491477 14501 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:48:22.491564 14501 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:48:22.491655 14497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:22.492422 14497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:22.499223 14497 catalog_manager.cc:1347] Generated new cluster ID: fdc1563f9107489ea0160f8646853277
I20260620 14:48:22.499297 14497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:22.507969 14497 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:22.508916 14497 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:22.514493 14497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5196d2ebe5d34eff970c28b43a64a941: Generated new TSK 0
I20260620 14:48:22.515165 14497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:48:22.534571 14484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:22.529177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45234 (local address 127.8.221.254:32857)
0620 14:48:22.529350 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:22.529358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:22.529564 (+ 206us) server_negotiation.cc:407] Connection header received
0620 14:48:22.529753 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:22.529755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:22.529861 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:22.529964 (+ 103us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:48:22.530256 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.530810 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.532107 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.532317 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.532397 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:22.532744 (+ 347us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:48:22.534206 (+ 1462us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38901/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:48:22.534433 (+ 227us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45234: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38901/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 14:48:22.534651 14484 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45234: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38901/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 14:48:22.536466 14510 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:22.529061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:32857 (local address 127.0.0.1:45234)
0620 14:48:22.529435 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:22.529462 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:22.529609 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:22.529989 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:22.529993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:22.530002 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:48:22.530133 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:22.530140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.530928 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.530931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:22.531986 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:22.531995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.532459 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.532463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:22.532545 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:22.532576 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:48:22.534388 (+ 1812us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38901/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:48:22.536237 (+ 1849us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.221.254:32857: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38901/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":1604,"thread_start_us":151,"threads_started":1}
I20260620 14:48:22.537245 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 14414
E20260620 14:48:22.545603 14511 webserver.cc:533] Webserver: sslize failed:
E20260620 14:48:22.545670 14511 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T14:48:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (322 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T14:48:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:22Z Disabled control of system clock
I20260620 14:48:22.587450 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:41945
--webserver_interface=127.8.221.254
--webserver_port=0
--builtin_ntp_servers=127.8.221.212:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:41945
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966871461349-9079-0/testchainca.cert with env {}
W20260620 14:48:22.703109 14515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:22.703467 14515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:22.703536 14515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:22.707660 14515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:22.707721 14515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:22.707737 14515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:22.707754 14515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:22.707770 14515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:48:22.707831 14515 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:48:22.707865 14515 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 14:48:22.713405 14515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:41945
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:41945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966871461349-9079-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:22.714545 14515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:22.715569 14515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:22.725232 14520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:22.725255 14521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:22.725344 14515 server_base.cc:1034] running on GCE node
W20260620 14:48:22.725232 14523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:22.726389 14515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:22.727860 14515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:22.729084 14515 hybrid_clock.cc:648] HybridClock initialized: now 1781966902729058 us; error 42 us; skew 500 ppm
I20260620 14:48:22.731706 14515 webserver.cc:460] Webserver started at http://127.8.221.254:35075/ using document root <none> and password file <none>
I20260620 14:48:22.732466 14515 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:22.732574 14515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:22.732852 14515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:22.735505 14515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "1d3ca39c203742848a39265d5c1c3bfa"
format_stamp: "Formatted at 2026-06-20 14:48:22 on dist-test-slave-65tz"
server_key: "e7dc645129fc161abf24fc1cf696c77e"
server_key_iv: "9fc9264f1938dc2d586edabf586b5768"
server_key_version: "encryptionkey@0"
I20260620 14:48:22.736229 14515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "1d3ca39c203742848a39265d5c1c3bfa"
format_stamp: "Formatted at 2026-06-20 14:48:22 on dist-test-slave-65tz"
server_key: "e7dc645129fc161abf24fc1cf696c77e"
server_key_iv: "9fc9264f1938dc2d586edabf586b5768"
server_key_version: "encryptionkey@0"
I20260620 14:48:22.741106 14515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 14:48:22.744302 14529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:22.745981 14515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:22.746138 14515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "1d3ca39c203742848a39265d5c1c3bfa"
format_stamp: "Formatted at 2026-06-20 14:48:22 on dist-test-slave-65tz"
server_key: "e7dc645129fc161abf24fc1cf696c77e"
server_key_iv: "9fc9264f1938dc2d586edabf586b5768"
server_key_version: "encryptionkey@0"
I20260620 14:48:22.746279 14515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:22.768368 14515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:22.769318 14515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:22.769553 14515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:22.778774 14515 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:41945
I20260620 14:48:22.779129 14581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:41945 every 8 connection(s)
I20260620 14:48:22.780611 14515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:22.783385 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 14515
I20260620 14:48:22.783526 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:48:22.783838 9079 external_mini_cluster.cc:1428] Setting key cdf64e7b03d63c30950ed636dcbced54
I20260620 14:48:22.785061 14582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:22.792047 14582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa: Bootstrap starting.
I20260620 14:48:22.794401 14582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:22.795251 14582 log.cc:824] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:22.796103 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:22.786228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.8.221.254:41945)
0620 14:48:22.787677 (+ 1449us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:22.787700 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:22.787747 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:48:22.788657 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:22.788679 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:22.789178 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:22.789584 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:22.790129 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.791203 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.792140 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.792754 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.792944 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:22.793957 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:22.793963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:22.793977 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:22.794444 (+ 467us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:22.794559 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:22.794591 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:22.794711 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:22.794850 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 14:48:22.795185 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":171,"threads_started":1}
I20260620 14:48:22.797396 14582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa: No bootstrap required, opened a new log
I20260620 14:48:22.799449 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:48:22.801405 14582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3ca39c203742848a39265d5c1c3bfa" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41945 } }
I20260620 14:48:22.801651 14582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:22.801726 14582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3ca39c203742848a39265d5c1c3bfa, State: Initialized, Role: FOLLOWER
I20260620 14:48:22.802205 14582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3ca39c203742848a39265d5c1c3bfa" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41945 } }
I20260620 14:48:22.802354 14582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:22.802438 14582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:22.802544 14582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:22.803544 14582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3ca39c203742848a39265d5c1c3bfa" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41945 } }
I20260620 14:48:22.803833 14582 leader_election.cc:302] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d3ca39c203742848a39265d5c1c3bfa; no voters:
I20260620 14:48:22.804081 14582 leader_election.cc:288] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:22.804257 14587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:22.804541 14587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa [term 1 LEADER]: Becoming Leader. State: Replica: 1d3ca39c203742848a39265d5c1c3bfa, State: Running, Role: LEADER
I20260620 14:48:22.804909 14587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3ca39c203742848a39265d5c1c3bfa" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41945 } }
I20260620 14:48:22.805364 14582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:22.807132 14589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d3ca39c203742848a39265d5c1c3bfa. Latest consensus state: current_term: 1 leader_uuid: "1d3ca39c203742848a39265d5c1c3bfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3ca39c203742848a39265d5c1c3bfa" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41945 } } }
I20260620 14:48:22.807271 14589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:22.807080 14588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d3ca39c203742848a39265d5c1c3bfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3ca39c203742848a39265d5c1c3bfa" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41945 } } }
I20260620 14:48:22.807523 14588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:22.807772 14601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:22.810825 14601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:22.816586 14601 catalog_manager.cc:1347] Generated new cluster ID: c3bf126ecce641e9b329ad2aea040cf4
I20260620 14:48:22.816660 14601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:22.821766 14601 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:22.823225 14601 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:22.832151 14601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d3ca39c203742848a39265d5c1c3bfa: Generated new TSK 0
I20260620 14:48:22.833149 14601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:48:22.917672 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:22.847731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58240 (local address 127.8.221.254:41945)
0620 14:48:22.847913 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:22.847919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:22.848069 (+ 150us) server_negotiation.cc:407] Connection header received
0620 14:48:22.848271 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:22.848274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:22.848325 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:22.848423 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:48:22.848777 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.849353 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.850752 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.851087 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.851238 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:22.851612 (+ 374us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:48:22.917044 (+ 65432us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:48:22.917475 (+ 431us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58240: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 14:48:22.917833 14585 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58240: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 14:48:22.917990 14611 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:22.847533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41945 (local address 127.0.0.1:58240)
0620 14:48:22.847918 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:22.847938 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:22.848119 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:22.848461 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:22.848464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:22.848474 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:48:22.848627 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:22.848633 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.849497 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.849501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:22.850605 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:22.850616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.851237 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.851241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:22.851344 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:22.851381 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:48:22.917433 (+ 66052us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:48:22.917689 (+ 256us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.221.254:41945: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":136,"threads_started":1}
I20260620 14:48:22.918764 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 14515
2026-06-20T14:48:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (361 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (148 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (132 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:23 dist-test-slave-65tz krb5kdc[14623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:23 dist-test-slave-65tz krb5kdc[14623](info): set up 2 sockets
Jun 20 14:48:23 dist-test-slave-65tz krb5kdc[14623](info): commencing operation
krb5kdc: starting...
W20260620 14:48:24.251593 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:48:24 dist-test-slave-65tz krb5kdc[14623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:48:24.387209 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:37923
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:37923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:24.519119 14636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:24.519430 14636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:24.519475 14636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:24.523700 14636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:24.523769 14636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:24.523790 14636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:24.523808 14636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:24.523823 14636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:24.529562 14636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:37923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:37923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:24.530975 14636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:24.532223 14636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:24.542034 14641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:24.542004 14642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:24.541989 14644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:24.542351 14636 server_base.cc:1034] running on GCE node
I20260620 14:48:24.543004 14636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:24.544215 14636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:24.545384 14636 hybrid_clock.cc:648] HybridClock initialized: now 1781966904545362 us; error 46 us; skew 500 ppm
Jun 20 14:48:24 dist-test-slave-65tz krb5kdc[14623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966904, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:24.548261 14636 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:48:24.549181 14636 webserver.cc:460] Webserver started at http://127.8.221.254:43749/ using document root <none> and password file <none>
I20260620 14:48:24.549784 14636 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:24.549926 14636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:24.550156 14636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:24.552266 14636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "b16a487c7f7d406dbab068f3264b6eea"
format_stamp: "Formatted at 2026-06-20 14:48:24 on dist-test-slave-65tz"
server_key: "c422ff7daee7c331a9dfda2ab0ab9daf"
server_key_iv: "501091fce8962056637c63fed33f798a"
server_key_version: "encryptionkey@0"
I20260620 14:48:24.552840 14636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "b16a487c7f7d406dbab068f3264b6eea"
format_stamp: "Formatted at 2026-06-20 14:48:24 on dist-test-slave-65tz"
server_key: "c422ff7daee7c331a9dfda2ab0ab9daf"
server_key_iv: "501091fce8962056637c63fed33f798a"
server_key_version: "encryptionkey@0"
I20260620 14:48:24.557086 14636 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 14:48:24.559872 14651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:24.561380 14636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:48:24.561553 14636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "b16a487c7f7d406dbab068f3264b6eea"
format_stamp: "Formatted at 2026-06-20 14:48:24 on dist-test-slave-65tz"
server_key: "c422ff7daee7c331a9dfda2ab0ab9daf"
server_key_iv: "501091fce8962056637c63fed33f798a"
server_key_version: "encryptionkey@0"
I20260620 14:48:24.561661 14636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:24.597329 14636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:24.600618 14636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:24.600824 14636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:24.608615 14636 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:37923
I20260620 14:48:24.608690 14703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:37923 every 8 connection(s)
I20260620 14:48:24.609619 14636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:24.613018 14704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:24.614630 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 14636
I20260620 14:48:24.614771 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:48:24.615049 9079 external_mini_cluster.cc:1428] Setting key ee08d55784cde91b83f5f0009a81b785
I20260620 14:48:24.620479 14704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea: Bootstrap starting.
I20260620 14:48:24.623423 14704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:24.624378 14704 log.cc:824] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:48:24 dist-test-slave-65tz krb5kdc[14623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:24.627170 14704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea: No bootstrap required, opened a new log
I20260620 14:48:24.630784 14707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:24.616674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.8.221.254:37923)
0620 14:48:24.617211 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:24.617228 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:24.617268 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:48:24.618163 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:24.618183 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:24.618551 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:24.618905 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:24.619696 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.620799 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.621649 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.622182 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.622356 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:24.625539 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:24.625544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:24.625558 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:24.625584 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:24.627728 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:24.628283 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:24.628287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:24.628292 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:24.628428 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:24.628734 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:24.628736 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:24.628737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:24.629122 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:24.629268 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:24.629620 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 14:48:24.629949 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":177,"threads_started":1}
I20260620 14:48:24.631906 14704 raft_consensus.cc:348] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16a487c7f7d406dbab068f3264b6eea" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 37923 } }
I20260620 14:48:24.632239 14704 raft_consensus.cc:374] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:24.632290 14704 raft_consensus.cc:729] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b16a487c7f7d406dbab068f3264b6eea, State: Initialized, Role: FOLLOWER
I20260620 14:48:24.632834 14704 consensus_queue.cc:260] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16a487c7f7d406dbab068f3264b6eea" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 37923 } }
I20260620 14:48:24.633005 14704 raft_consensus.cc:388] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:24.633062 14704 raft_consensus.cc:482] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:24.633178 14704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:24.634384 14704 raft_consensus.cc:504] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16a487c7f7d406dbab068f3264b6eea" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 37923 } }
I20260620 14:48:24.634791 14704 leader_election.cc:302] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b16a487c7f7d406dbab068f3264b6eea; no voters:
I20260620 14:48:24.635197 14704 leader_election.cc:288] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:24.635353 14709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:24.635656 14709 raft_consensus.cc:686] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea [term 1 LEADER]: Becoming Leader. State: Replica: b16a487c7f7d406dbab068f3264b6eea, State: Running, Role: LEADER
I20260620 14:48:24.636053 14709 consensus_queue.cc:237] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16a487c7f7d406dbab068f3264b6eea" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 37923 } }
I20260620 14:48:24.636744 14704 sys_catalog.cc:564] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:24.637987 14711 sys_catalog.cc:455] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea [sys.catalog]: SysCatalogTable state changed. Reason: New leader b16a487c7f7d406dbab068f3264b6eea. Latest consensus state: current_term: 1 leader_uuid: "b16a487c7f7d406dbab068f3264b6eea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16a487c7f7d406dbab068f3264b6eea" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 37923 } } }
I20260620 14:48:24.638149 14711 sys_catalog.cc:458] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:24.638500 14710 sys_catalog.cc:455] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b16a487c7f7d406dbab068f3264b6eea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16a487c7f7d406dbab068f3264b6eea" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 37923 } } }
I20260620 14:48:24.638576 14718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:24.638630 14710 sys_catalog.cc:458] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:24.641906 14718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:24.649168 14718 catalog_manager.cc:1347] Generated new cluster ID: fd6751ad2a014e408f2171c702176b19
I20260620 14:48:24.649255 14718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:24.669687 14718 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:24.670850 14718 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:24.678385 14718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b16a487c7f7d406dbab068f3264b6eea: Generated new TSK 0
I20260620 14:48:24.679068 14718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
I20260620 14:48:24.720293 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:37923
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:24.865650 14730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:24.866060 14730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:24.866120 14730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:24.870529 14730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:24.870609 14730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:24.870679 14730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:48:24.876613 14730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:37923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:24.878093 14730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:24.879400 14730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:24.891217 14736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:24.891194 14735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:24.891194 14738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:24.891259 14730 server_base.cc:1034] running on GCE node
I20260620 14:48:24.891866 14730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:24.892769 14730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:24.894013 14730 hybrid_clock.cc:648] HybridClock initialized: now 1781966904893981 us; error 61 us; skew 500 ppm
Jun 20 14:48:24 dist-test-slave-65tz krb5kdc[14623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966904, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:24.897115 14730 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:48:24.898185 14730 webserver.cc:460] Webserver started at http://127.8.221.193:33267/ using document root <none> and password file <none>
I20260620 14:48:24.898778 14730 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:24.898841 14730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:24.899065 14730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:24.901145 14730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "4c3af372abce41488e9f6c83b9235dbc"
format_stamp: "Formatted at 2026-06-20 14:48:24 on dist-test-slave-65tz"
server_key: "0560dcf2ef3697ad8fe86ca4a8cf6f6d"
server_key_iv: "74e27a0fe2871d51024470b75bad9abd"
server_key_version: "encryptionkey@0"
I20260620 14:48:24.901715 14730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "4c3af372abce41488e9f6c83b9235dbc"
format_stamp: "Formatted at 2026-06-20 14:48:24 on dist-test-slave-65tz"
server_key: "0560dcf2ef3697ad8fe86ca4a8cf6f6d"
server_key_iv: "74e27a0fe2871d51024470b75bad9abd"
server_key_version: "encryptionkey@0"
I20260620 14:48:24.905701 14730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:24.908531 14745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:24.910060 14730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:24.910202 14730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "4c3af372abce41488e9f6c83b9235dbc"
format_stamp: "Formatted at 2026-06-20 14:48:24 on dist-test-slave-65tz"
server_key: "0560dcf2ef3697ad8fe86ca4a8cf6f6d"
server_key_iv: "74e27a0fe2871d51024470b75bad9abd"
server_key_version: "encryptionkey@0"
I20260620 14:48:24.910331 14730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:24.931890 14730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:24.935186 14730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:24.935388 14730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:24.935993 14730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:24.936983 14730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:24.937198 14730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:24.937316 14730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:24.937371 14730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:24.947777 14730 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:43801
I20260620 14:48:24.947847 14858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:43801 every 8 connection(s)
I20260620 14:48:24.948819 14730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:24.958738 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 14730
I20260620 14:48:24.958890 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:48:24.959174 9079 external_mini_cluster.cc:1428] Setting key 2f4af6d8c51cbd87a5c2468e82e54547
Jun 20 14:48:24 dist-test-slave-65tz krb5kdc[14623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966904, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:24.965050 14707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:24.951125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:44471 (local address 127.8.221.254:37923)
0620 14:48:24.951293 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:24.951297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:24.952166 (+ 869us) server_negotiation.cc:407] Connection header received
0620 14:48:24.953757 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:24.953760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:24.953846 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:24.953947 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:24.955370 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.955918 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.957216 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.957454 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.957584 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:24.960939 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:24.960943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:24.960946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:24.960972 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:24.962816 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:24.963374 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:24.963378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:24.963381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:24.963485 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:24.963826 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:24.963831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:24.963833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:24.964017 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:24.964128 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:24.964772 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 14:48:24.964905 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:48:24.966269 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:24.951437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:37923 (local address 127.8.221.193:44471)
0620 14:48:24.951941 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:24.951992 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:24.953528 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:24.954187 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:24.954195 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:24.954621 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:24.955219 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:24.955234 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.956041 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.956043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:24.957084 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:24.957100 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.957677 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.957684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:24.957880 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:24.958637 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:24.958655 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:24.960748 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:24.962953 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:24.962958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:24.962971 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:24.963251 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:24.963587 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:24.963592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:24.963594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:24.963730 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:24.964122 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:24.964128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:24.964457 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 14:48:24.964700 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 14:48:24.965103 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":108,"threads_started":1}
I20260620 14:48:24.967752 14859 heartbeater.cc:347] Connected to a master server at 127.8.221.254:37923
I20260620 14:48:24.968082 14859 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:24.968763 14859 heartbeater.cc:511] Master 127.8.221.254:37923 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.221.194@KRBTEST.COM; defaulting to no policy
I20260620 14:48:24.970582 14668 ts_manager.cc:194] Registered new tserver with Master: 4c3af372abce41488e9f6c83b9235dbc (127.8.221.193:43801)
I20260620 14:48:24.971968 14668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:44471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
I20260620 14:48:24.989414 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:37923
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:25.106931 14864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:25.107195 14864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:25.107237 14864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:25.111136 14864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:25.111186 14864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:25.111241 14864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:48:25.116629 14864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:37923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:25.117650 14864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:25.118654 14864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:25.128778 14870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:25.128765 14872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:25.128765 14869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:25.129570 14864 server_base.cc:1034] running on GCE node
I20260620 14:48:25.130324 14864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:25.131053 14864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:25.132268 14864 hybrid_clock.cc:648] HybridClock initialized: now 1781966905132229 us; error 63 us; skew 500 ppm
Jun 20 14:48:25 dist-test-slave-65tz krb5kdc[14623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966905, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:25.135241 14864 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:48:25.136142 14864 webserver.cc:460] Webserver started at http://127.8.221.194:35195/ using document root <none> and password file <none>
I20260620 14:48:25.136664 14864 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:25.136725 14864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:25.136917 14864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:25.139129 14864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "db4b207114ed4a05a52a7f7305daa69e"
format_stamp: "Formatted at 2026-06-20 14:48:25 on dist-test-slave-65tz"
server_key: "242952363103d153c91554d85fe2f4c2"
server_key_iv: "469c13aeca774b2fb6c941270ffeabde"
server_key_version: "encryptionkey@0"
I20260620 14:48:25.139700 14864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "db4b207114ed4a05a52a7f7305daa69e"
format_stamp: "Formatted at 2026-06-20 14:48:25 on dist-test-slave-65tz"
server_key: "242952363103d153c91554d85fe2f4c2"
server_key_iv: "469c13aeca774b2fb6c941270ffeabde"
server_key_version: "encryptionkey@0"
I20260620 14:48:25.143806 14864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 14:48:25.146540 14879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:25.147948 14864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:25.148085 14864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "db4b207114ed4a05a52a7f7305daa69e"
format_stamp: "Formatted at 2026-06-20 14:48:25 on dist-test-slave-65tz"
server_key: "242952363103d153c91554d85fe2f4c2"
server_key_iv: "469c13aeca774b2fb6c941270ffeabde"
server_key_version: "encryptionkey@0"
I20260620 14:48:25.148208 14864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:25.159453 14864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:25.162267 14864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:25.162458 14864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:25.163021 14864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:25.163826 14864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:25.163895 14864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:25.163960 14864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:25.164000 14864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:25.173786 14864 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:45729
I20260620 14:48:25.173894 14992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:45729 every 8 connection(s)
I20260620 14:48:25.174698 14864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:25.176435 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 14864
I20260620 14:48:25.176540 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:25.176851 9079 external_mini_cluster.cc:1428] Setting key 0e03781c1b29fb79e33f7ef275c8dee8
WARNING: no policy specified for kudu/127.8.221.195@KRBTEST.COM; defaulting to no policy
Jun 20 14:48:25 dist-test-slave-65tz krb5kdc[14623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966905, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.195@KRBTEST.COM" created.
I20260620 14:48:25.191146 14707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:25.176988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:45637 (local address 127.8.221.254:37923)
0620 14:48:25.177201 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:25.177207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:25.178262 (+ 1055us) server_negotiation.cc:407] Connection header received
0620 14:48:25.179955 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:25.179959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:25.180026 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:25.180112 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:25.181491 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.182205 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.183443 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.183678 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.183794 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:25.186912 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:25.186916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:25.186919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:25.186949 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:25.188528 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:25.189234 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:25.189239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:25.189241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:25.189350 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:25.189733 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:25.189739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:25.189741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:25.190025 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:25.190195 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:25.190775 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 14:48:25.190938 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 14:48:25.192158 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:25.177412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:37923 (local address 127.8.221.194:45637)
0620 14:48:25.178069 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:25.178139 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:25.179697 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:25.180327 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:25.180334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:25.180771 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:25.181342 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:25.181357 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.182346 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.182350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:25.183315 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:25.183324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.183840 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.183845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:25.184040 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:25.184786 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:25.184803 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:25.186687 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:25.188674 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:25.188680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:25.188695 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:25.189051 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:25.189462 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:25.189465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:25.189467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:25.189629 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:25.190187 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:25.190194 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:25.190499 (+ 305us) client_negotiation.cc:769] Sending connection context
0620 14:48:25.190732 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 14:48:25.191125 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":153,"threads_started":1}
I20260620 14:48:25.193658 14993 heartbeater.cc:347] Connected to a master server at 127.8.221.254:37923
I20260620 14:48:25.193989 14993 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:25.194594 14993 heartbeater.cc:511] Master 127.8.221.254:37923 requested a full tablet report, sending...
I20260620 14:48:25.195868 14668 ts_manager.cc:194] Registered new tserver with Master: db4b207114ed4a05a52a7f7305daa69e (127.8.221.194:45729)
I20260620 14:48:25.196576 14668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:45637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
I20260620 14:48:25.204959 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:37923
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:45611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:25.313411 14998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:25.313668 14998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:25.313706 14998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:25.317973 14998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:25.318044 14998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:25.318102 14998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:48:25.323379 14998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:45611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:37923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:25.324421 14998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:25.325495 14998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:25.335839 15003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:25.335839 15006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:25.335882 15004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:25.336511 14998 server_base.cc:1034] running on GCE node
I20260620 14:48:25.337177 14998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:25.338193 14998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:25.339372 14998 hybrid_clock.cc:648] HybridClock initialized: now 1781966905339350 us; error 38 us; skew 500 ppm
Jun 20 14:48:25 dist-test-slave-65tz krb5kdc[14623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966905, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:25.346216 14998 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:48:25.347383 14998 webserver.cc:460] Webserver started at http://127.8.221.195:35697/ using document root <none> and password file <none>
I20260620 14:48:25.348107 14998 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:25.348204 14998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:25.348476 14998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:25.351018 14998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "a5b7c739de3a4d74a0d945aec5a80180"
format_stamp: "Formatted at 2026-06-20 14:48:25 on dist-test-slave-65tz"
server_key: "835a79f4ccf3b21f59a77c2cae4e4280"
server_key_iv: "c1d48716100075048843690c51cb8f0e"
server_key_version: "encryptionkey@0"
I20260620 14:48:25.351753 14998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "a5b7c739de3a4d74a0d945aec5a80180"
format_stamp: "Formatted at 2026-06-20 14:48:25 on dist-test-slave-65tz"
server_key: "835a79f4ccf3b21f59a77c2cae4e4280"
server_key_iv: "c1d48716100075048843690c51cb8f0e"
server_key_version: "encryptionkey@0"
I20260620 14:48:25.356398 14998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:48:25.359465 15013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:25.360997 14998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:48:25.361157 14998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "a5b7c739de3a4d74a0d945aec5a80180"
format_stamp: "Formatted at 2026-06-20 14:48:25 on dist-test-slave-65tz"
server_key: "835a79f4ccf3b21f59a77c2cae4e4280"
server_key_iv: "c1d48716100075048843690c51cb8f0e"
server_key_version: "encryptionkey@0"
I20260620 14:48:25.361306 14998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:25.385147 14998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:25.388839 14998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:25.389251 14998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:25.390214 14998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:25.391340 14998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:25.391419 14998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:25.391503 14998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:25.391558 14998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:25.403959 14998 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:45629
I20260620 14:48:25.403998 15126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:45629 every 8 connection(s)
I20260620 14:48:25.405133 14998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:25.411643 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 14998
I20260620 14:48:25.411798 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:25.412119 9079 external_mini_cluster.cc:1428] Setting key a97053dee6d99835738d5606846468aa
Jun 20 14:48:25 dist-test-slave-65tz krb5kdc[14623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966905, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:25.421326 14707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:25.407602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:41721 (local address 127.8.221.254:37923)
0620 14:48:25.407837 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:25.407843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:25.408876 (+ 1033us) server_negotiation.cc:407] Connection header received
0620 14:48:25.410426 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:25.410429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:25.410487 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:25.410595 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:25.412097 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.412872 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.414237 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.414634 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.414760 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:25.417459 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:25.417462 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:25.417464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:25.417487 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:25.418997 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:25.419501 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:25.419503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:25.419505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:25.419576 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:25.420051 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:25.420055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:25.420057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:25.420415 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:25.420525 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:25.421009 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 14:48:25.421144 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 14:48:25.422150 15129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:25.408140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:37923 (local address 127.8.221.195:41721)
0620 14:48:25.408695 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:25.408743 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:25.410191 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:25.410771 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:25.410779 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:25.411175 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:25.411904 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:25.411924 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.413009 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.413012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:25.414118 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:25.414127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.414860 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.414866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:25.415034 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:25.415730 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:25.415747 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:25.417286 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:25.419148 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:25.419152 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:25.419162 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:25.419410 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:25.419704 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:25.419707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:25.419709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:25.419897 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:25.420534 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:25.420543 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:25.420777 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 14:48:25.420968 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 14:48:25.421277 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":134,"threads_started":1}
I20260620 14:48:25.423614 15127 heartbeater.cc:347] Connected to a master server at 127.8.221.254:37923
I20260620 14:48:25.423794 15127 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:25.424387 15127 heartbeater.cc:511] Master 127.8.221.254:37923 requested a full tablet report, sending...
I20260620 14:48:25.425879 14668 ts_manager.cc:194] Registered new tserver with Master: a5b7c739de3a4d74a0d945aec5a80180 (127.8.221.195:45629)
I20260620 14:48:25.426865 14668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:41721
I20260620 14:48:25.427498 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/test-admin.keytab.
Jun 20 14:48:25 dist-test-slave-65tz krb5kdc[14623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:25.447435 9079 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 14:48:25.451004 15134 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:25.449146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.8.221.193:43801)
0620 14:48:25.449509 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:25.449532 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:25.449568 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:48:25.449663 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:25.449670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:25.450006 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:25.450223 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:25.450751 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54360: BlockingRecv error: recv got EOF from 127.0.0.1:54360 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":88,"threads_started":1}
W20260620 14:48:25.451103 9079 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:48:25 dist-test-slave-65tz krb5kdc[14623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:25.452490 9079 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:48:25 dist-test-slave-65tz krb5kdc[14623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:25.453619 9079 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:48:25 dist-test-slave-65tz krb5kdc[14623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.193@KRBTEST.COM
I20260620 14:48:25.464233 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:25.454648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54368 (local address 127.8.221.193:43801)
0620 14:48:25.454778 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:25.454783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:25.454912 (+ 129us) server_negotiation.cc:407] Connection header received
0620 14:48:25.455018 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:25.455020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:25.455057 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:25.455129 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:25.455643 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.456489 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.457384 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.457786 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.457953 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:25.460338 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:25.460348 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:25.460365 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:25.460392 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:25.462233 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:25.462709 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:25.462715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:25.462717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:25.462800 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:25.463101 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:25.463103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:25.463104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:25.463384 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:25.463555 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:25.463826 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 14:48:25.464073 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 14:48:25.468818 15134 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:25.467699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54380 (local address 127.8.221.193:43801)
0620 14:48:25.467879 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:25.467889 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:25.467970 (+ 81us) server_negotiation.cc:407] Connection header received
0620 14:48:25.468069 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:25.468071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:25.468118 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:25.468202 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:25.468690 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54380: BlockingRecv error: recv got EOF from 127.0.0.1:54380 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 14:48:25.468905 9079 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:48:25 dist-test-slave-65tz krb5kdc[14623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:25.470115 9079 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:48:25 dist-test-slave-65tz krb5kdc[14623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:25.471076 9079 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:48:25 dist-test-slave-65tz krb5kdc[14623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.193@KRBTEST.COM
I20260620 14:48:25.480333 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:25.471891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54384 (local address 127.8.221.193:43801)
0620 14:48:25.472032 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:25.472035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:25.472078 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:48:25.472177 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:25.472180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:25.472216 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:25.472288 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:25.472747 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.473380 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.474303 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.474555 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.474646 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:25.477028 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:25.477032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:25.477034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:25.477057 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:25.478645 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:25.479066 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:25.479069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:25.479071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:25.479177 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:25.479436 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:25.479438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:25.479440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:25.479614 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:25.479723 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:25.479958 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 14:48:25.480152 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 14:48:25.483490 15134 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:25.482328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54392 (local address 127.8.221.193:43801)
0620 14:48:25.482617 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:25.482620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:25.482634 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:48:25.482667 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:25.482669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:25.482707 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:25.482783 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:25.483349 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54392: BlockingRecv error: recv got EOF from 127.0.0.1:54392 (error 108)
Metrics: {"server-negotiator.queue_time_us":180}
W20260620 14:48:25.483711 9079 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966871461349-9079-0/krb5kdc/krb5cc)
Jun 20 14:48:25 dist-test-slave-65tz krb5kdc[14623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:25.484802 9079 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:48:25 dist-test-slave-65tz krb5kdc[14623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:25.485708 9079 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:48:25 dist-test-slave-65tz krb5kdc[14623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.193@KRBTEST.COM
I20260620 14:48:25.495028 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:25.486623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.8.221.193:43801)
0620 14:48:25.486788 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:25.486795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:25.486880 (+ 85us) server_negotiation.cc:407] Connection header received
0620 14:48:25.487021 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:25.487025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:25.487078 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:25.487164 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:25.487660 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.488195 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.488980 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.489221 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.489338 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:25.491702 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:25.491705 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:25.491707 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:25.491727 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:25.493244 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:25.493756 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:25.493759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:25.493760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:25.493889 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:25.494308 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:25.494311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:25.494312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:25.494461 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:25.494566 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:25.494804 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 14:48:25.494904 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:48:25.496187 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 14730
I20260620 14:48:25.502090 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 14864
I20260620 14:48:25.507642 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 14998
I20260620 14:48:25.513231 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 14636
2026-06-20T14:48:25Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2315 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:25 dist-test-slave-65tz krb5kdc[15147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:25 dist-test-slave-65tz krb5kdc[15147](info): set up 2 sockets
Jun 20 14:48:25 dist-test-slave-65tz krb5kdc[15147](info): commencing operation
krb5kdc: starting...
W20260620 14:48:26.591540 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:48:26 dist-test-slave-65tz krb5kdc[15147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.254.keytab.
Entry for principal oryx/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.254.keytab.
I20260620 14:48:26.733665 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:41225
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.254.keytab
--principal=oryx/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:37325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:41225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:26.848239 15160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:26.848515 15160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:26.848559 15160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:26.852599 15160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:26.852664 15160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:26.852684 15160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:26.852702 15160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:26.852718 15160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:26.858253 15160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:37325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:41225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.254.keytab
--principal=oryx/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:41225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:26.859330 15160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:26.860338 15160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:26.869964 15166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:26.869983 15165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:26.869992 15168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:26.870504 15160 server_base.cc:1034] running on GCE node
I20260620 14:48:26.871268 15160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:26.872797 15160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:26.874035 15160 hybrid_clock.cc:648] HybridClock initialized: now 1781966906873989 us; error 67 us; skew 500 ppm
Jun 20 14:48:26 dist-test-slave-65tz krb5kdc[15147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966906, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:26.877615 15160 init.cc:377] Logged in from keytab as oryx/127.8.221.254@KRBTEST.COM (short username oryx)
I20260620 14:48:26.878904 15160 webserver.cc:460] Webserver started at http://127.8.221.254:44091/ using document root <none> and password file <none>
I20260620 14:48:26.879653 15160 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:26.879766 15160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:26.880062 15160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:26.882869 15160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "d3c4ab27b63c43d1b12d4e179d350d98"
format_stamp: "Formatted at 2026-06-20 14:48:26 on dist-test-slave-65tz"
server_key: "65b2e9d086cae1ab6be21134107235d1"
server_key_iv: "56328528fb0414fdbfc631a7487ee9df"
server_key_version: "encryptionkey@0"
I20260620 14:48:26.883594 15160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "d3c4ab27b63c43d1b12d4e179d350d98"
format_stamp: "Formatted at 2026-06-20 14:48:26 on dist-test-slave-65tz"
server_key: "65b2e9d086cae1ab6be21134107235d1"
server_key_iv: "56328528fb0414fdbfc631a7487ee9df"
server_key_version: "encryptionkey@0"
I20260620 14:48:26.888682 15160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:48:26.891984 15175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:26.893743 15160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:26.894047 15160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "d3c4ab27b63c43d1b12d4e179d350d98"
format_stamp: "Formatted at 2026-06-20 14:48:26 on dist-test-slave-65tz"
server_key: "65b2e9d086cae1ab6be21134107235d1"
server_key_iv: "56328528fb0414fdbfc631a7487ee9df"
server_key_version: "encryptionkey@0"
I20260620 14:48:26.894207 15160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:26.945657 15160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:26.949713 15160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:26.950018 15160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:26.958650 15160 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:41225
I20260620 14:48:26.959049 15227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:41225 every 8 connection(s)
I20260620 14:48:26.959882 15160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:26.962076 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 15160
I20260620 14:48:26.962198 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:48:26.962558 9079 external_mini_cluster.cc:1428] Setting key 4f98c3faace0cb8141c83b1e3a581ffb
I20260620 14:48:26.963331 15228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:26.970742 15228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98: Bootstrap starting.
I20260620 14:48:26.973441 15228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98: Neither blocks nor log segments found. Creating new log.
Jun 20 14:48:26 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM
I20260620 14:48:26.974352 15228 log.cc:824] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:26.977085 15228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98: No bootstrap required, opened a new log
I20260620 14:48:26.980165 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:26.965038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.8.221.254:41225)
0620 14:48:26.965540 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:26.965569 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:26.965612 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:48:26.966564 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:26.966585 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:26.967030 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:26.967440 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:26.968386 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:26.969630 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:26.970794 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:26.971440 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:26.971638 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:26.975031 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:26.975040 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:26.975059 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:26.975097 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:26.977028 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:26.977596 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:26.977601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:26.977605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:26.977739 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:26.978167 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:26.978171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:26.978177 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:26.978616 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:26.978808 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:26.979164 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 14:48:26.979427 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":168,"threads_started":1}
I20260620 14:48:26.982143 15228 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } }
I20260620 14:48:26.982486 15228 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:26.982550 15228 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c4ab27b63c43d1b12d4e179d350d98, State: Initialized, Role: FOLLOWER
I20260620 14:48:26.983111 15228 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } }
I20260620 14:48:26.983287 15228 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:26.983381 15228 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:26.983490 15228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:26.984746 15228 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } }
I20260620 14:48:26.985317 15228 leader_election.cc:302] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3c4ab27b63c43d1b12d4e179d350d98; no voters:
I20260620 14:48:26.985931 15228 leader_election.cc:288] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:26.986123 15233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:26.986439 15233 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [term 1 LEADER]: Becoming Leader. State: Replica: d3c4ab27b63c43d1b12d4e179d350d98, State: Running, Role: LEADER
I20260620 14:48:26.986828 15233 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } }
I20260620 14:48:26.987576 15228 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:26.988660 15235 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c4ab27b63c43d1b12d4e179d350d98. Latest consensus state: current_term: 1 leader_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } }
I20260620 14:48:26.988854 15235 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:26.988911 15234 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } }
I20260620 14:48:26.989030 15234 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:26.989616 15242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:26.992911 15242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:26.998425 15242 catalog_manager.cc:1347] Generated new cluster ID: 1337fddfcbd84c00afdea2e078edfce1
I20260620 14:48:26.998493 15242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:27.014544 15242 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:27.015566 15242 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:27.031355 15242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98: Generated new TSK 0
I20260620 14:48:27.032051 15242 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.193.keytab.
Entry for principal oryx/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.193.keytab.
I20260620 14:48:27.073434 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:41225
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.193.keytab
--principal=oryx/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:37325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:27.233321 15254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:27.233636 15254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:27.233765 15254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:27.238440 15254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:27.238526 15254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:27.238597 15254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:48:27.244831 15254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:37325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.193.keytab
--principal=oryx/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:27.246273 15254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:27.247520 15254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:27.259230 15260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:27.259230 15262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:27.259253 15259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:27.259670 15254 server_base.cc:1034] running on GCE node
I20260620 14:48:27.260272 15254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:27.260991 15254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:27.262154 15254 hybrid_clock.cc:648] HybridClock initialized: now 1781966907262139 us; error 37 us; skew 500 ppm
Jun 20 14:48:27 dist-test-slave-65tz krb5kdc[15147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966907, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:27.265161 15254 init.cc:377] Logged in from keytab as oryx/127.8.221.193@KRBTEST.COM (short username oryx)
I20260620 14:48:27.266193 15254 webserver.cc:460] Webserver started at http://127.8.221.193:36711/ using document root <none> and password file <none>
I20260620 14:48:27.266820 15254 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:27.266885 15254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:27.267105 15254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:27.269236 15254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "6b5d90216c0747ecbf9f808e728294ac"
format_stamp: "Formatted at 2026-06-20 14:48:27 on dist-test-slave-65tz"
server_key: "3060f6fc2e739f3f541fea2ff90e2a36"
server_key_iv: "22c087036d38a811678e4838b318109d"
server_key_version: "encryptionkey@0"
I20260620 14:48:27.269861 15254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "6b5d90216c0747ecbf9f808e728294ac"
format_stamp: "Formatted at 2026-06-20 14:48:27 on dist-test-slave-65tz"
server_key: "3060f6fc2e739f3f541fea2ff90e2a36"
server_key_iv: "22c087036d38a811678e4838b318109d"
server_key_version: "encryptionkey@0"
I20260620 14:48:27.273861 15254 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:27.276458 15269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:27.277954 15254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:27.278079 15254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "6b5d90216c0747ecbf9f808e728294ac"
format_stamp: "Formatted at 2026-06-20 14:48:27 on dist-test-slave-65tz"
server_key: "3060f6fc2e739f3f541fea2ff90e2a36"
server_key_iv: "22c087036d38a811678e4838b318109d"
server_key_version: "encryptionkey@0"
I20260620 14:48:27.278180 15254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:27.301563 15254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:27.304826 15254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:27.305024 15254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:27.305624 15254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:27.306519 15254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:27.306589 15254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:27.306663 15254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:27.306702 15254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:27.316802 15254 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:41493
I20260620 14:48:27.316836 15382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:41493 every 8 connection(s)
I20260620 14:48:27.317730 15254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:27.323107 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 15254
I20260620 14:48:27.323212 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:48:27.323534 9079 external_mini_cluster.cc:1428] Setting key 1a4adcd60459b5157e35c005d324001c
Jun 20 14:48:27 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966907, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.8.221.194@KRBTEST.COM; defaulting to no policy
I20260620 14:48:27.334854 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:27.319987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:33149 (local address 127.8.221.254:41225)
0620 14:48:27.320183 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:27.320189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:27.321078 (+ 889us) server_negotiation.cc:407] Connection header received
0620 14:48:27.322507 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:27.322510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:27.322565 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:27.322649 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:27.324416 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.325038 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.326377 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.326797 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.326918 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:27.330322 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:27.330329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:27.330335 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:27.330368 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:27.332346 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:27.332917 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:27.332921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:27.332923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:27.333038 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:27.333421 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:27.333425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:27.333428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:27.333682 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:27.333913 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:27.334583 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 14:48:27.334721 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 14:48:27.336019 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:27.320353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.193:33149)
0620 14:48:27.320850 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:27.320901 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:27.322279 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:27.322841 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:27.322848 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:27.323445 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:27.324248 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:27.324268 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.325168 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.325172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:27.326251 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:27.326261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.326989 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.326995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:27.327229 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:27.327987 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:27.328009 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:27.330136 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:27.332483 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:27.332488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:27.332502 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:27.332780 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:27.333152 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:27.333155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:27.333156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:27.333297 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:27.333869 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:27.333878 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:27.334249 (+ 371us) client_negotiation.cc:769] Sending connection context
0620 14:48:27.334488 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 14:48:27.334819 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.221.194@KRBTEST.COM" created.
I20260620 14:48:27.337594 15383 heartbeater.cc:347] Connected to a master server at 127.8.221.254:41225
I20260620 14:48:27.337954 15383 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:27.338681 15383 heartbeater.cc:511] Master 127.8.221.254:41225 requested a full tablet report, sending...
I20260620 14:48:27.340767 15192 ts_manager.cc:194] Registered new tserver with Master: 6b5d90216c0747ecbf9f808e728294ac (127.8.221.193:41493)
I20260620 14:48:27.342497 15192 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:33149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.194.keytab.
Entry for principal oryx/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.194.keytab.
I20260620 14:48:27.354341 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:41225
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.194.keytab
--principal=oryx/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:37325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:27.473071 15388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:27.473338 15388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:27.473380 15388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:27.477366 15388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:27.477422 15388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:27.477478 15388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:48:27.483016 15388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:37325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.194.keytab
--principal=oryx/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:27.484033 15388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:27.485005 15388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:27.495906 15393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:27.495939 15394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:27.496099 15388 server_base.cc:1034] running on GCE node
W20260620 14:48:27.496289 15396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:27.496675 15388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:27.497296 15388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:27.498476 15388 hybrid_clock.cc:648] HybridClock initialized: now 1781966907498435 us; error 56 us; skew 500 ppm
Jun 20 14:48:27 dist-test-slave-65tz krb5kdc[15147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966907, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:27.501940 15388 init.cc:377] Logged in from keytab as oryx/127.8.221.194@KRBTEST.COM (short username oryx)
I20260620 14:48:27.502985 15388 webserver.cc:460] Webserver started at http://127.8.221.194:43589/ using document root <none> and password file <none>
I20260620 14:48:27.503648 15388 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:27.503716 15388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:27.503986 15388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:27.506377 15388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b"
format_stamp: "Formatted at 2026-06-20 14:48:27 on dist-test-slave-65tz"
server_key: "961c730d0ee3a151665f7c27f11cd4e7"
server_key_iv: "0a9f076f3c69314b2fc2883b0da225ae"
server_key_version: "encryptionkey@0"
I20260620 14:48:27.507011 15388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b"
format_stamp: "Formatted at 2026-06-20 14:48:27 on dist-test-slave-65tz"
server_key: "961c730d0ee3a151665f7c27f11cd4e7"
server_key_iv: "0a9f076f3c69314b2fc2883b0da225ae"
server_key_version: "encryptionkey@0"
I20260620 14:48:27.511365 15388 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:48:27.514427 15403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:27.516149 15388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:48:27.516266 15388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b"
format_stamp: "Formatted at 2026-06-20 14:48:27 on dist-test-slave-65tz"
server_key: "961c730d0ee3a151665f7c27f11cd4e7"
server_key_iv: "0a9f076f3c69314b2fc2883b0da225ae"
server_key_version: "encryptionkey@0"
I20260620 14:48:27.516420 15388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:27.540771 15388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:27.544543 15388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:27.544777 15388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:27.545513 15388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:27.546578 15388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:27.546633 15388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:27.546705 15388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:27.546739 15388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:27.557976 15388 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:38715
I20260620 14:48:27.558082 15516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:38715 every 8 connection(s)
I20260620 14:48:27.559060 15388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:27.561538 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 15388
I20260620 14:48:27.561666 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:27.562022 9079 external_mini_cluster.cc:1428] Setting key bc36592724c98b7b4c75560ddb36fecd
Jun 20 14:48:27 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966907, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.8.221.195@KRBTEST.COM; defaulting to no policy
I20260620 14:48:27.575757 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:27.561141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:49873 (local address 127.8.221.254:41225)
0620 14:48:27.561274 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:27.561278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:27.562430 (+ 1152us) server_negotiation.cc:407] Connection header received
0620 14:48:27.564070 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:27.564073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:27.564122 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:27.564201 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:27.565877 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.566844 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.568047 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.568468 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.568609 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:27.571505 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:27.571509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:27.571513 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:27.571547 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:27.573359 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:27.574006 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:27.574009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:27.574010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:27.574091 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:27.574483 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:27.574486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:27.574487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:27.574713 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:27.574828 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:27.575370 (+ 542us) server_negotiation.cc:299] Negotiation successful
0620 14:48:27.575509 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.221.195@KRBTEST.COM" created.
I20260620 14:48:27.576671 15519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:27.561575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.194:49873)
0620 14:48:27.562222 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:27.562287 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:27.563826 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:27.564394 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:27.564406 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:27.564963 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:27.565697 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:27.565713 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.566992 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.566995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:27.567915 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:27.567924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.568733 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.568740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:27.568931 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:27.569597 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:27.569622 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:27.571275 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:27.573486 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:27.573490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:27.573504 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:27.573778 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:27.574196 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:27.574199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:27.574200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:27.574330 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:27.574860 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:27.574866 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:27.575092 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 14:48:27.575268 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 14:48:27.575536 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":170,"threads_started":1}
I20260620 14:48:27.578066 15517 heartbeater.cc:347] Connected to a master server at 127.8.221.254:41225
I20260620 14:48:27.578457 15517 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:27.579288 15517 heartbeater.cc:511] Master 127.8.221.254:41225 requested a full tablet report, sending...
I20260620 14:48:27.580797 15192 ts_manager.cc:194] Registered new tserver with Master: d3c00e694fc84e3eaf83fd9ec6a87d0b (127.8.221.194:38715)
I20260620 14:48:27.581535 15192 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:49873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.195.keytab.
Entry for principal oryx/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.195.keytab.
I20260620 14:48:27.595323 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:41225
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.195.keytab
--principal=oryx/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:37325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:27.725317 15522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:27.725643 15522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:27.725690 15522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:27.729810 15522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:27.729875 15522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:27.729939 15522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:48:27.735563 15522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:37325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.195.keytab
--principal=oryx/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:41225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:27.736909 15522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:27.738209 15522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:27.749500 15528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:27.749501 15527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:27.749500 15530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:27.750048 15522 server_base.cc:1034] running on GCE node
I20260620 14:48:27.750622 15522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:27.751415 15522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:27.752640 15522 hybrid_clock.cc:648] HybridClock initialized: now 1781966907752599 us; error 56 us; skew 500 ppm
Jun 20 14:48:27 dist-test-slave-65tz krb5kdc[15147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966907, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:27.755602 15522 init.cc:377] Logged in from keytab as oryx/127.8.221.195@KRBTEST.COM (short username oryx)
I20260620 14:48:27.756490 15522 webserver.cc:460] Webserver started at http://127.8.221.195:45893/ using document root <none> and password file <none>
I20260620 14:48:27.757091 15522 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:27.757156 15522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:27.757334 15522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:27.759562 15522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "113fb3b62558486f8265f5bc4311d8a2"
format_stamp: "Formatted at 2026-06-20 14:48:27 on dist-test-slave-65tz"
server_key: "d161557c9aadc1b6a07c79fed97e19cd"
server_key_iv: "aa7d6e19d00f93d57fd47c2f2e9285c8"
server_key_version: "encryptionkey@0"
I20260620 14:48:27.760089 15522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "113fb3b62558486f8265f5bc4311d8a2"
format_stamp: "Formatted at 2026-06-20 14:48:27 on dist-test-slave-65tz"
server_key: "d161557c9aadc1b6a07c79fed97e19cd"
server_key_iv: "aa7d6e19d00f93d57fd47c2f2e9285c8"
server_key_version: "encryptionkey@0"
I20260620 14:48:27.764236 15522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:48:27.766984 15537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:27.768515 15522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:27.768630 15522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "113fb3b62558486f8265f5bc4311d8a2"
format_stamp: "Formatted at 2026-06-20 14:48:27 on dist-test-slave-65tz"
server_key: "d161557c9aadc1b6a07c79fed97e19cd"
server_key_iv: "aa7d6e19d00f93d57fd47c2f2e9285c8"
server_key_version: "encryptionkey@0"
I20260620 14:48:27.768788 15522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:27.784750 15522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:27.788004 15522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:27.788210 15522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:27.788877 15522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:27.789749 15522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:27.789873 15522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:27.789940 15522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:27.789968 15522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:27.800314 15522 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:34701
I20260620 14:48:27.800359 15650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:34701 every 8 connection(s)
I20260620 14:48:27.801273 15522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:27.803690 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 15522
I20260620 14:48:27.803821 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:27.804113 9079 external_mini_cluster.cc:1428] Setting key fb4b7f56b087eb9c8a5653d4f35433e7
Jun 20 14:48:27 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966907, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM
I20260620 14:48:27.816228 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:27.803550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:55735 (local address 127.8.221.254:41225)
0620 14:48:27.803768 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:27.803774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:27.804876 (+ 1102us) server_negotiation.cc:407] Connection header received
0620 14:48:27.806422 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:27.806424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:27.806471 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:27.806544 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:27.807826 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.808356 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.809568 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.809782 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.809933 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:27.812745 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:27.812752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:27.812755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:27.812779 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:27.814160 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:27.814709 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:27.814714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:27.814716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:27.814795 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:27.815137 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:27.815140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:27.815142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:27.815327 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:27.815434 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:27.815940 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 14:48:27.816055 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 14:48:27.817194 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:27.804080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.195:55735)
0620 14:48:27.804693 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:27.804754 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:27.806217 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:27.806702 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:27.806708 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:27.807129 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:27.807694 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:27.807709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.808469 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.808471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:27.809456 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:27.809466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.809990 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.809993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:27.810198 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:27.810961 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:27.810978 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:27.812566 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:27.814316 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:27.814322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:27.814338 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:27.814605 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:27.814888 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:27.814892 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:27.814894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:27.815033 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:27.815431 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:27.815438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:27.815703 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 14:48:27.815904 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 14:48:27.816259 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":177,"threads_started":1}
I20260620 14:48:27.818585 15651 heartbeater.cc:347] Connected to a master server at 127.8.221.254:41225
I20260620 14:48:27.818840 15651 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:27.819336 15651 heartbeater.cc:511] Master 127.8.221.254:41225 requested a full tablet report, sending...
I20260620 14:48:27.820405 15192 ts_manager.cc:194] Registered new tserver with Master: 113fb3b62558486f8265f5bc4311d8a2 (127.8.221.195:34701)
I20260620 14:48:27.821064 15192 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:55735
I20260620 14:48:27.830196 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.221.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.221.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.221.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.253.keytab.
Entry for principal oryx/127.8.221.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.253.keytab.
I20260620 14:48:27.858636 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.253:37131
--webserver_interface=127.8.221.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.253.keytab
--principal=oryx/127.8.221.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:37325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:41225,127.8.221.253:37131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:27.988168 15656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:27.988566 15656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:27.988674 15656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:27.993351 15656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:27.993507 15656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:27.993562 15656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:27.993618 15656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:27.993657 15656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:28.003350 15656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:37325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:41225,127.8.221.253:37131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.253.keytab
--principal=oryx/127.8.221.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.253:37131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.221.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
W20260620 14:48:28.003737 15656 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.8.221.254:41225,127.8.221.253:37131'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 14:48:28.005239 15656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:28.007047 15656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:28.021251 15661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:28.021270 15664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:28.021270 15662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:28.021716 15656 server_base.cc:1034] running on GCE node
I20260620 14:48:28.022675 15656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:28.024612 15656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:28.025864 15656 hybrid_clock.cc:648] HybridClock initialized: now 1781966908025777 us; error 101 us; skew 500 ppm
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966908, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:28.030536 15656 init.cc:377] Logged in from keytab as oryx/127.8.221.253@KRBTEST.COM (short username oryx)
I20260620 14:48:28.031981 15656 webserver.cc:460] Webserver started at http://127.8.221.253:43007/ using document root <none> and password file <none>
I20260620 14:48:28.032867 15656 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:28.032971 15656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:28.033288 15656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:28.036492 15656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/data/instance:
uuid: "3e72bddb92f142bb9e99468a591005e0"
format_stamp: "Formatted at 2026-06-20 14:48:28 on dist-test-slave-65tz"
server_key: "945d055f188dac09af69db4c73436819"
server_key_iv: "11ee4d2fab9e106ff543b3732966eb21"
server_key_version: "encryptionkey@0"
I20260620 14:48:28.037122 15656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/wal/instance:
uuid: "3e72bddb92f142bb9e99468a591005e0"
format_stamp: "Formatted at 2026-06-20 14:48:28 on dist-test-slave-65tz"
server_key: "945d055f188dac09af69db4c73436819"
server_key_iv: "11ee4d2fab9e106ff543b3732966eb21"
server_key_version: "encryptionkey@0"
I20260620 14:48:28.042763 15656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:48:28.046543 15671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:28.048488 15656 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:48:28.048629 15656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/wal
uuid: "3e72bddb92f142bb9e99468a591005e0"
format_stamp: "Formatted at 2026-06-20 14:48:28 on dist-test-slave-65tz"
server_key: "945d055f188dac09af69db4c73436819"
server_key_iv: "11ee4d2fab9e106ff543b3732966eb21"
server_key_version: "encryptionkey@0"
I20260620 14:48:28.048758 15656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:28.091795 15656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:28.094981 15656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:28.095207 15656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:28.103154 15656 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.253:37131
I20260620 14:48:28.103513 15723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.253:37131 every 8 connection(s)
I20260620 14:48:28.104249 15656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/data/info.pb
I20260620 14:48:28.106385 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 15656
I20260620 14:48:28.108111 15724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:28.113646 15724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.221.253@KRBTEST.COM
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966908, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.253@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM
I20260620 14:48:28.125195 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.108145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45356 (local address 127.8.221.253:37131)
0620 14:48:28.108668 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.108690 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.108743 (+ 53us) server_negotiation.cc:407] Connection header received
0620 14:48:28.109731 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.109755 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.110274 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.110736 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.111819 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.113407 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.114612 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.115291 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.115481 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.119203 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.119210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.119220 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.119258 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.121631 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.122234 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.122241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.122247 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.122400 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.122719 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.122724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.122726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.123214 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.123474 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.123715 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.124053 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":124,"threads_started":1}
I20260620 14:48:28.127231 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.116068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53080 (local address 127.8.221.254:41225)
0620 14:48:28.116223 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.116228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.116727 (+ 499us) server_negotiation.cc:407] Connection header received
0620 14:48:28.116910 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.116912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.116956 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.117034 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.118050 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.118798 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.120023 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.120405 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.120530 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.123313 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.123318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.123325 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.123350 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.125039 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.125654 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.125657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.125659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.125749 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.126135 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.126138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.126140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.126427 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.126530 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.127030 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.127125 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:48:28.127450 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.115990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:53080)
0620 14:48:28.116558 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.116618 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.116771 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.117130 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.117138 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.117654 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.117919 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.117927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.118945 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.118949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.119879 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.119897 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.120546 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.120552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.120676 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.121238 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.121254 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.123167 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.125170 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.125177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.125182 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.125561 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.125939 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.125942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.125943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.126030 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.126561 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.126568 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.126703 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.126967 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.127172 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":172,"threads_started":1}
I20260620 14:48:28.128947 15724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966908, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.253@KRBTEST.COM for oryx/127.8.221.253@KRBTEST.COM
I20260620 14:48:28.138262 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.129566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:37131 (local address 127.0.0.1:45368)
0620 14:48:28.129756 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.129777 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.129895 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.130186 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.130189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.130383 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.130473 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.130479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.131683 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.131686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.132207 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.132214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.132698 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.132702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.132846 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.133291 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.133303 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.134700 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.136605 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.136608 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.136610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.136828 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.137194 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.137197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.137198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.137340 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.137764 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.137766 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.137914 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.138007 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.138120 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":39}
I20260620 14:48:28.138429 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.129664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.8.221.253:37131)
0620 14:48:28.129779 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.129785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.129841 (+ 56us) server_negotiation.cc:407] Connection header received
0620 14:48:28.130001 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.130006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.130085 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.130178 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.130609 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.131487 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.132346 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.132575 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.132689 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.134850 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.134855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.134864 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.134896 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.136481 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.136976 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.136979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.136981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.137070 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.137463 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.137467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.137468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.137664 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.137759 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.138088 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.138196 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":54}
I20260620 14:48:28.141672 15724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: Bootstrap starting.
I20260620 14:48:28.143936 15724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:28.144644 15724 log.cc:824] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:28.146976 15724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: No bootstrap required, opened a new log
I20260620 14:48:28.150086 15724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } }
I20260620 14:48:28.150338 15724 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:28.150420 15724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e72bddb92f142bb9e99468a591005e0, State: Initialized, Role: FOLLOWER
I20260620 14:48:28.150861 15724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } }
I20260620 14:48:28.151490 15730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } } }
I20260620 14:48:28.151721 15730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:48:28.152192 15724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:48:28.157745 15741 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:48:28.157907 15741 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:48:28.159876 15656 master_runner.cc:428] Detected that this master 3e72bddb92f142bb9e99468a591005e0 is joining an existing cluster
I20260620 14:48:28.159953 15656 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.221.253:37131
I20260620 14:48:28.171730 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.162885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45378 (local address 127.8.221.253:37131)
0620 14:48:28.163004 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.163008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.163026 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:48:28.163176 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.163179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.163234 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.163346 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.164058 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.164885 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.166083 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.166305 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.166442 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.167868 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.167872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.167875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.167906 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.169718 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.170292 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.170296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.170298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.170400 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.170736 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.170739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.170741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.171016 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.171126 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.171368 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.171515 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":46,"spinlock_wait_cycles":768}
I20260620 14:48:28.171751 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.162436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:37131 (local address 127.0.0.1:45378)
0620 14:48:28.162891 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.162908 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.163030 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.163366 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.163369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.163695 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.163906 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.163929 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.165020 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.165023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.165963 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.165971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.166453 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.166456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.166558 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.167183 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.167204 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.167729 (+ 525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.169874 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.169878 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.169880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.170168 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.170513 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.170516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.170518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.170617 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.171112 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.171115 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.171207 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.171317 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.171603 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":181,"thread_start_us":96,"threads_started":1}
I20260620 14:48:28.172370 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.162512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53084 (local address 127.8.221.254:41225)
0620 14:48:28.162705 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.162710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.163084 (+ 374us) server_negotiation.cc:407] Connection header received
0620 14:48:28.163250 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.163253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.163326 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.163438 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.164092 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.164660 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.166075 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.166471 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.166591 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.168441 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.168447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.168453 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.168490 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.170473 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.170950 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.170954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.170956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.171071 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.171381 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.171384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.171386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.171610 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.171685 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.172054 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.172198 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:48:28.173326 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.162450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:53084)
0620 14:48:28.162981 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.162993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.163098 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.163585 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.163588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.163817 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.163929 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.163935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.164781 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.164786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.165930 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.165941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.166749 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.166754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.166875 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.167358 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.167374 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.168011 (+ 637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.170560 (+ 2549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.170565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.170571 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.170831 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.171191 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.171194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.171197 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.171284 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.171770 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.171773 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.171867 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.172962 (+ 1095us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.173121 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":83,"thread_start_us":136,"threads_started":1}
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966906, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.254@KRBTEST.COM for oryx/127.8.221.253@KRBTEST.COM
WARNING: no policy specified for oryx/127.8.221.252@KRBTEST.COM; defaulting to no policy
I20260620 14:48:28.191030 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.178410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45394 (local address 127.8.221.253:37131)
0620 14:48:28.178546 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.178551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.179126 (+ 575us) server_negotiation.cc:407] Connection header received
0620 14:48:28.179400 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.179403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.179461 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.179552 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.180345 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.181154 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.182539 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.182959 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.183037 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.186370 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.186374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.186378 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.186404 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.188276 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.189101 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.189105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.189107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.189192 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.189580 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.189587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.189590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.189994 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.190069 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.190702 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.190840 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:48:28.191064 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.178343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:37131 (local address 127.0.0.1:45394)
0620 14:48:28.178998 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.179051 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.179222 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.179603 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.179609 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.180064 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.180199 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.180206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.181262 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.181266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.182400 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.182411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.183195 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.183199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.183306 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.183986 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.184001 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.186195 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.188429 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.188446 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.188455 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.188912 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.189305 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.189308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.189310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.189423 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.190244 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.190250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.190383 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.190625 (+ 242us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.190892 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":128,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.221.252@KRBTEST.COM" created.
I20260620 14:48:28.193009 15192 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.221.253:37131: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3e72bddb92f142bb9e99468a591005e0"
member_type: NON_VOTER
last_known_addr {
host: "127.8.221.253"
port: 37131
}
attrs {
promote: true
}
}
dest_uuid: "d3c4ab27b63c43d1b12d4e179d350d98"
cas_config_opid_index: -1
I20260620 14:48:28.193787 15192 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: NON_VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: true } }
I20260620 14:48:28.196291 15753 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3e72bddb92f142bb9e99468a591005e0 (127.8.221.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: NON_VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: true } } }
I20260620 14:48:28.197290 15698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:28.198616 15753 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.221.253:37131
I20260620 14:48:28.198983 15698 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 1 FOLLOWER]: Refusing update from remote peer d3c4ab27b63c43d1b12d4e179d350d98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 14:48:28.199173 15656 master.cc:537] Master@127.8.221.253:37131 shutting down...
I20260620 14:48:28.199038 15753 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: NON_VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: true } } }
I20260620 14:48:28.199417 15753 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:28.200409 15753 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: NON_VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:48:28.203349 15177 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 14:48:28.204557 15177 consensus_peers.cc:487] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 -> Peer 3e72bddb92f142bb9e99468a591005e0 (127.8.221.253:37131): Couldn't send request to peer 3e72bddb92f142bb9e99468a591005e0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:48:28.205392 15656 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:28.205742 15656 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:28.205904 15656 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.221.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.252.keytab.
Entry for principal oryx/127.8.221.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.252.keytab.
I20260620 14:48:28.213505 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.252:37817
--webserver_interface=127.8.221.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.252.keytab
--principal=oryx/127.8.221.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:37325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:41225,127.8.221.253:37131,127.8.221.252:37817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:48:28.221360 15656 master.cc:552] Master@127.8.221.253:37131 shutdown complete.
I20260620 14:48:28.221446 15656 master_runner.cc:305] Clearing existing system tablet
I20260620 14:48:28.221764 15656 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:28.221925 15656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:28.223606 15656 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:28.224682 15759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:28.225055 15656 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:28.225234 15656 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:28.225273 15656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/wal
uuid: "3e72bddb92f142bb9e99468a591005e0"
format_stamp: "Formatted at 2026-06-20 14:48:28 on dist-test-slave-65tz"
server_key: "945d055f188dac09af69db4c73436819"
server_key_iv: "11ee4d2fab9e106ff543b3732966eb21"
server_key_version: "encryptionkey@0"
I20260620 14:48:28.228152 15656 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:28.231096 15656 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:48:28.231202 15656 log.cc:1196] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 14:48:28.231505 15656 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: Deleting consensus metadata
I20260620 14:48:28.231700 15656 master_runner.cc:315] Copying system tablet from 127.8.221.254:41225
I20260620 14:48:28.233129 15656 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: tablet copy: Beginning tablet copy session from remote peer at address 127.8.221.254:41225
I20260620 14:48:28.243842 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.233839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:53094)
0620 14:48:28.234234 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.234251 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.234381 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.234766 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.234772 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.235004 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.235164 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.235171 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.236621 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.236626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.237663 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.237673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.238360 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.238363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.238461 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.239151 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.239165 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.239562 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.241959 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.241962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.241964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.242289 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.242636 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.242639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.242641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.242735 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.243262 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.243265 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.243367 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.243486 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.243630 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":108,"threads_started":1}
I20260620 14:48:28.243921 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.234029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53094 (local address 127.8.221.254:41225)
0620 14:48:28.234193 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.234199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.234354 (+ 155us) server_negotiation.cc:407] Connection header received
0620 14:48:28.234541 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.234544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.234610 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.234716 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.235323 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.236459 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.237833 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.238223 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.238346 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.239732 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.239736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.239740 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.239768 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.241833 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.242407 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.242411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.242414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.242522 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.242844 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.242848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.242850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.243146 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.243262 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.243576 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.243715 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:48:28.244988 15212 tablet_copy_service.cc:140] P d3c4ab27b63c43d1b12d4e179d350d98: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3e72bddb92f142bb9e99468a591005e0 ({username='oryx', principal='oryx/127.8.221.253@KRBTEST.COM'} at 127.0.0.1:53094)
I20260620 14:48:28.245221 15212 tablet_copy_service.cc:161] P d3c4ab27b63c43d1b12d4e179d350d98: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3e72bddb92f142bb9e99468a591005e0 at {username='oryx', principal='oryx/127.8.221.253@KRBTEST.COM'} at 127.0.0.1:53094: session id = 3e72bddb92f142bb9e99468a591005e0-00000000000000000000000000000000
I20260620 14:48:28.247434 15212 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:48:28.249449 15656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:48:28.252173 15656 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: tablet copy: Starting download of 0 data blocks...
I20260620 14:48:28.252409 15656 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: tablet copy: Starting download of 1 WAL segments...
I20260620 14:48:28.254652 15656 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:48:28.256735 15212 tablet_copy_service.cc:342] P d3c4ab27b63c43d1b12d4e179d350d98: Request end of tablet copy session 3e72bddb92f142bb9e99468a591005e0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.221.253@KRBTEST.COM'} at 127.0.0.1:53094
I20260620 14:48:28.256944 15212 tablet_copy_service.cc:433] P d3c4ab27b63c43d1b12d4e179d350d98: ending tablet copy session 3e72bddb92f142bb9e99468a591005e0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3e72bddb92f142bb9e99468a591005e0
I20260620 14:48:28.266042 15656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:28.268329 15766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:28.268591 15767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:28.270987 15769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:28.271188 15656 server_base.cc:1034] running on GCE node
I20260620 14:48:28.271381 15656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:28.271773 15656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:28.272925 15656 hybrid_clock.cc:648] HybridClock initialized: now 1781966908272907 us; error 38 us; skew 500 ppm
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966908, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:28.275035 15656 init.cc:377] Logged in from keytab as oryx/127.8.221.253@KRBTEST.COM (short username oryx)
I20260620 14:48:28.275604 15656 webserver.cc:460] Webserver started at http://127.8.221.253:33827/ using document root <none> and password file <none>
I20260620 14:48:28.275836 15656 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:28.275888 15656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:28.277465 15656 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:48:28.278725 15776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:28.279148 15656 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:48:28.279239 15656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/wal
uuid: "3e72bddb92f142bb9e99468a591005e0"
format_stamp: "Formatted at 2026-06-20 14:48:28 on dist-test-slave-65tz"
server_key: "945d055f188dac09af69db4c73436819"
server_key_iv: "11ee4d2fab9e106ff543b3732966eb21"
server_key_version: "encryptionkey@0"
I20260620 14:48:28.279351 15656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:28.333714 15656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:28.336426 15656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:28.342316 15656 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.253:37131
I20260620 14:48:28.342450 15828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.253:37131 every 8 connection(s)
I20260620 14:48:28.343004 15656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-1/data/info.pb
I20260620 14:48:28.345000 15829 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:48:28.345566 15383 heartbeater.cc:503] Master 127.8.221.254:41225 was elected leader, sending a full tablet report...
I20260620 14:48:28.346271 15829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: Bootstrap starting.
I20260620 14:48:28.357311 15829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:48:28.357622 15829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: Bootstrap complete.
I20260620 14:48:28.358227 15829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: NON_VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: true } }
I20260620 14:48:28.358373 15829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e72bddb92f142bb9e99468a591005e0, State: Initialized, Role: LEARNER
I20260620 14:48:28.358487 15829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: NON_VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: true } }
I20260620 14:48:28.358935 15829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:28.358861 15835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: NON_VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: true } } }
I20260620 14:48:28.359177 15835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [sys.catalog]: This master's current role is: LEARNER
W20260620 14:48:28.360540 15757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:28.360929 15757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:28.361016 15757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:48:28.362937 15846 catalog_manager.cc:1259] Loaded cluster ID: 1337fddfcbd84c00afdea2e078edfce1
I20260620 14:48:28.363072 15846 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: loading cluster ID for follower catalog manager: success
I20260620 14:48:28.365433 15846 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: acquiring CA information for follower catalog manager: success
I20260620 14:48:28.367177 15846 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 14:48:28.367384 15757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:28.367475 15757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:28.367548 15757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:28.367600 15757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:28.367637 15757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:28.373721 15757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:37325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:41225,127.8.221.253:37131,127.8.221.252:37817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.252.keytab
--principal=oryx/127.8.221.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.252:37817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.221.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:28.374887 15757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:28.375988 15757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:28.386745 15851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:28.386756 15848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:28.386756 15849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:28.387158 15757 server_base.cc:1034] running on GCE node
I20260620 14:48:28.387734 15757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:28.389015 15757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:28.390182 15757 hybrid_clock.cc:648] HybridClock initialized: now 1781966908390155 us; error 41 us; skew 500 ppm
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966908, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:28.393170 15757 init.cc:377] Logged in from keytab as oryx/127.8.221.252@KRBTEST.COM (short username oryx)
I20260620 14:48:28.394153 15757 webserver.cc:460] Webserver started at http://127.8.221.252:41893/ using document root <none> and password file <none>
I20260620 14:48:28.394707 15757 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:28.394778 15757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:28.394986 15757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:28.397009 15757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/data/instance:
uuid: "914c6ea4107a41b18d22d611649e61e2"
format_stamp: "Formatted at 2026-06-20 14:48:28 on dist-test-slave-65tz"
server_key: "cb731df68d1078d344bf94bee570526f"
server_key_iv: "75657f5ec7c876db6f4cac7a58fb9021"
server_key_version: "encryptionkey@0"
I20260620 14:48:28.397559 15757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/wal/instance:
uuid: "914c6ea4107a41b18d22d611649e61e2"
format_stamp: "Formatted at 2026-06-20 14:48:28 on dist-test-slave-65tz"
server_key: "cb731df68d1078d344bf94bee570526f"
server_key_iv: "75657f5ec7c876db6f4cac7a58fb9021"
server_key_version: "encryptionkey@0"
I20260620 14:48:28.401614 15757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:28.404167 15858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:28.405592 15757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:28.405745 15757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/wal
uuid: "914c6ea4107a41b18d22d611649e61e2"
format_stamp: "Formatted at 2026-06-20 14:48:28 on dist-test-slave-65tz"
server_key: "cb731df68d1078d344bf94bee570526f"
server_key_iv: "75657f5ec7c876db6f4cac7a58fb9021"
server_key_version: "encryptionkey@0"
I20260620 14:48:28.405901 15757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:28.414966 15757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:28.418074 15757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:28.418306 15757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:28.426154 15757 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.252:37817
I20260620 14:48:28.426501 15910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.252:37817 every 8 connection(s)
I20260620 14:48:28.427106 15757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/data/info.pb
I20260620 14:48:28.430444 15911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:28.432363 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 15757
I20260620 14:48:28.435534 15911 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.221.252@KRBTEST.COM
I20260620 14:48:28.449985 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.433857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45956 (local address 127.8.221.252:37817)
0620 14:48:28.434317 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.434346 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.434396 (+ 50us) server_negotiation.cc:407] Connection header received
0620 14:48:28.435345 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.435375 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.435856 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.436355 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.437764 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.439345 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.440486 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.441161 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.441670 (+ 509us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.444351 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.444356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.444374 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.444404 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.446807 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.447307 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.447312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.447316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.447418 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.447728 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.447731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.447732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.448214 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.448408 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.448796 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.449069 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":106,"threads_started":1}
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966908, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.252@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM
I20260620 14:48:28.455251 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.437764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:53104)
0620 14:48:28.445562 (+ 7798us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.445618 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.445743 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.446230 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.446238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.446675 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.446839 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.446849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.447958 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.447963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.448904 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.448916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.449411 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.449416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.449540 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.450239 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.450251 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.451786 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.453684 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.453691 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.453695 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.453992 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.454283 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.454286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.454287 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.454381 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.454746 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.454750 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.454847 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.454958 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.455099 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7581,"thread_start_us":182,"threads_started":1}
I20260620 14:48:28.455442 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.445358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53104 (local address 127.8.221.254:41225)
0620 14:48:28.445917 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.445921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.445942 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:48:28.446003 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.446007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.446063 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.446167 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.447014 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.447822 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.449066 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.449288 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.449414 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.451935 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.451939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.451942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.451968 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.453573 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.454105 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.454108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.454109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.454185 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.454486 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.454488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.454489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.454656 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.454734 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.455103 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.455212 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409}
I20260620 14:48:28.456617 15911 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966908, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.252@KRBTEST.COM for oryx/127.8.221.253@KRBTEST.COM
I20260620 14:48:28.465559 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.457246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:37131 (local address 127.0.0.1:45406)
0620 14:48:28.457400 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.457414 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.457476 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.458026 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.458029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.458174 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.458288 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.458294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.459348 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.459351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.460155 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.460164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.460655 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.460660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.460782 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.461261 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.461272 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.462572 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.464162 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.464165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.464166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.464356 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.464681 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.464683 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.464684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.464757 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.465157 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.465159 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.465234 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.465322 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.465429 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 14:48:28.465664 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.457418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45406 (local address 127.8.221.253:37131)
0620 14:48:28.457718 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.457729 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.457741 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:48:28.457773 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.457775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.457885 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.457959 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.458444 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.459219 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.460283 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.460521 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.460605 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.462703 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.462706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.462708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.462727 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.464055 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.464475 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.464478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.464479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.464569 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.464868 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.464871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.464872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.465046 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.465148 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.465389 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.465490 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":111,"threads_started":1}
I20260620 14:48:28.466549 15911 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966908, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.252@KRBTEST.COM for oryx/127.8.221.252@KRBTEST.COM
I20260620 14:48:28.475178 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.467148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37817 (local address 127.0.0.1:45958)
0620 14:48:28.467270 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.467283 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.467388 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.467621 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.467624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.467783 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.467871 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.467877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.469035 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.469038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.469515 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.469522 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.470046 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.470049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.470163 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.470678 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.470690 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.471977 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.473642 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.473648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.473652 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.473910 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.474181 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.474184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.474186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.474262 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.474638 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.474641 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.474717 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.474846 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.474976 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":20}
I20260620 14:48:28.475178 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.467272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.8.221.252:37817)
0620 14:48:28.467413 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.467420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.467435 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:48:28.467493 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.467500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.467531 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.467612 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.467983 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.468503 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.469629 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.469939 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.470049 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.472106 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.472110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.472112 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.472133 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.473545 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.474008 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.474011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.474013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.474090 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.474368 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.474374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.474377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.474545 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.474651 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.474889 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.475028 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":80}
I20260620 14:48:28.478785 15911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: Bootstrap starting.
I20260620 14:48:28.481695 15911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:28.482602 15911 log.cc:824] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:28.484956 15911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: No bootstrap required, opened a new log
I20260620 14:48:28.489380 15911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } }
I20260620 14:48:28.489612 15911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:28.489682 15911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914c6ea4107a41b18d22d611649e61e2, State: Initialized, Role: FOLLOWER
I20260620 14:48:28.490219 15911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } }
I20260620 14:48:28.490840 15917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } } }
I20260620 14:48:28.491057 15917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:48:28.491711 15911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:48:28.496654 15928 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:48:28.496750 15928 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:48:28.500744 15757 master_runner.cc:428] Detected that this master 914c6ea4107a41b18d22d611649e61e2 is joining an existing cluster
I20260620 14:48:28.500823 15757 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.221.252:37817
I20260620 14:48:28.503801 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 15254
I20260620 14:48:28.513900 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.503903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.8.221.254:41225)
0620 14:48:28.504052 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.504059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.504689 (+ 630us) server_negotiation.cc:407] Connection header received
0620 14:48:28.504897 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.504900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.504963 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.505017 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.505748 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.506696 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.508018 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.508371 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.508453 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.510238 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.510243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.510246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.510271 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.512005 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.512499 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.512503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.512505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.512586 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.512907 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.512914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.512916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.513127 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.513200 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.513537 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.513658 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:48:28.514434 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.503418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:53114)
0620 14:48:28.504596 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.504612 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.504760 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.505188 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.505191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.505441 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.505607 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.505616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.506853 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.506857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.507873 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.507883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.508650 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.508653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.508751 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.509520 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.509541 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.510069 (+ 528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.512127 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.512130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.512133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.512391 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.512696 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.512700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.512702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.512789 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.513271 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.513274 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.513363 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.514073 (+ 710us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.514235 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"thread_start_us":132,"threads_started":1}
I20260620 14:48:28.514786 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:41493
--local_ip_for_outbound_sockets=127.8.221.193
--tserver_master_addrs=127.8.221.254:41225
--webserver_port=36711
--webserver_interface=127.8.221.193
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.193.keytab
--principal=oryx/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:37325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.221.254:41225,127.8.221.253:37131
--tserver_master_addrs=127.8.221.254:41225,127.8.221.253:37131,127.8.221.252:37817 with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:48:28.515705 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.503901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45422 (local address 127.8.221.253:37131)
0620 14:48:28.504682 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.504689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.506168 (+ 1479us) server_negotiation.cc:407] Connection header received
0620 14:48:28.506348 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.506354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.506413 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.506469 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.507241 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.508154 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.509580 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.510051 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.510143 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.511727 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.511733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.511740 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.511769 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.513576 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.514199 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.514203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.514205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.514286 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.514596 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.514613 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.514614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.514825 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.514904 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.515310 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.515451 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687}
I20260620 14:48:28.516404 15934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.503492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:37131 (local address 127.0.0.1:45422)
0620 14:48:28.506077 (+ 2585us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.506094 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.506241 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.506644 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.506650 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.506934 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.507104 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.507113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.508298 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.508304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.509446 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.509460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.510314 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.510320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.510439 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.511007 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.511026 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.511562 (+ 536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.513701 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.513707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.513712 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.514093 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.514405 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.514409 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.514411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.514493 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.514993 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.514998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.515123 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.515896 (+ 773us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.516119 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2493,"thread_start_us":102,"threads_started":1}
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966906, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.254@KRBTEST.COM for oryx/127.8.221.252@KRBTEST.COM
I20260620 14:48:28.539104 15936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.504178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37817 (local address 127.0.0.1:45974)
0620 14:48:28.513992 (+ 9814us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.514010 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.514142 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.514472 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.514477 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.514706 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.514820 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.514828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.515722 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.515725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.516522 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.516531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.517092 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.517095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.517154 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.534306 (+ 17152us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.534322 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.534852 (+ 530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.537221 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.537224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.537226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.537609 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.537991 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.537994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.537996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.538082 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.538504 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.538507 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.538599 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.538690 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.538862 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9740,"spinlock_wait_cycles":4480,"thread_start_us":75,"threads_started":1}
I20260620 14:48:28.539709 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.505148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.8.221.252:37817)
0620 14:48:28.505276 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.505283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.514092 (+ 8809us) server_negotiation.cc:407] Connection header received
0620 14:48:28.514232 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.514234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.514272 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.514352 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.514942 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.515606 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.516642 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.516938 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.516995 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.535015 (+ 18020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.535019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.535022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.535043 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.537105 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.537709 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.537712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.537714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.537787 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.538166 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.538169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.538171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.538390 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.538474 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.539460 (+ 986us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.539576 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:48:28.539875 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.528376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37817 (local address 127.0.0.1:45986)
0620 14:48:28.528571 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.528585 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.528682 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.529135 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.529138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.529351 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.529475 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.529481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.530674 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.530677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.531594 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.531611 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.532218 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.532222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.532320 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.532927 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.532941 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.534609 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.538080 (+ 3471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.538083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.538086 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.538405 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.538728 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.538731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.538733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.538818 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.539285 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.539289 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.539385 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.539461 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.539647 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 14:48:28.540817 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.528472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45986 (local address 127.8.221.252:37817)
0620 14:48:28.528802 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.528821 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.528838 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:48:28.528928 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.528931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.528992 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.529104 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.529603 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.530550 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.531728 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.532096 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.532206 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.534790 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.534794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.534797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.534826 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.537873 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.538531 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.538534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.538536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.538629 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.538921 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.538925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.538927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.539164 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.540179 (+ 1015us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.540336 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.540470 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":136,"threads_started":1}
I20260620 14:48:28.542114 15192 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.221.252:37817: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "914c6ea4107a41b18d22d611649e61e2"
member_type: NON_VOTER
last_known_addr {
host: "127.8.221.252"
port: 37817
}
attrs {
promote: true
}
}
dest_uuid: "d3c4ab27b63c43d1b12d4e179d350d98"
cas_config_opid_index: 5
I20260620 14:48:28.542937 15192 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: NON_VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: true } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: NON_VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: true } }
I20260620 14:48:28.544382 15753 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 914c6ea4107a41b18d22d611649e61e2 (127.8.221.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: NON_VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: true } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: NON_VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: true } } }
I20260620 14:48:28.546310 15753 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.221.252:37817
I20260620 14:48:28.546530 15753 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: NON_VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: true } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: NON_VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: true } } }
I20260620 14:48:28.546636 15753 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:28.546801 15757 master.cc:537] Master@127.8.221.252:37817 shutting down...
W20260620 14:48:28.549000 15178 consensus_peers.cc:487] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 -> Peer 914c6ea4107a41b18d22d611649e61e2 (127.8.221.252:37817): Couldn't send request to peer 914c6ea4107a41b18d22d611649e61e2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:48:28.553865 15757 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:28.554222 15757 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:28.554266 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.544340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45438 (local address 127.8.221.253:37131)
0620 14:48:28.544455 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.544460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.549151 (+ 4691us) server_negotiation.cc:407] Connection header received
0620 14:48:28.549224 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.549227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.549308 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.549419 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:28.549831 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.550658 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.552333 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.552967 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.553067 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.553203 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.554067 (+ 864us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.554141 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:48:28.554337 15757 tablet_replica.cc:330] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: stopping tablet replica
I20260620 14:48:28.554764 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.544244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:37131 (local address 127.0.0.1:45438)
0620 14:48:28.548840 (+ 4596us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.548857 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.548959 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.549483 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.549486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.549514 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:28.549632 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.549639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.550961 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.550965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.552191 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.552202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.553512 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.553516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.553634 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.553782 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.553885 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.554647 (+ 762us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4487}
I20260620 14:48:28.555431 15803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 1 LEARNER]: Refusing update from remote peer d3c4ab27b63c43d1b12d4e179d350d98: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:48:28.556205 15939 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: NON_VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 14:48:28.559244 15803 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 914c6ea4107a41b18d22d611649e61e2 (127.8.221.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: NON_VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: true } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: NON_VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: true } } }
I20260620 14:48:28.561928 15835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c4ab27b63c43d1b12d4e179d350d98. Latest consensus state: current_term: 1 leader_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: NON_VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: true } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: NON_VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: true } } }
I20260620 14:48:28.562089 15835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:48:28.561996 15941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: NON_VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: true } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: NON_VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: true } } }
I20260620 14:48:28.562258 15941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:48:28.565599 15757 master.cc:552] Master@127.8.221.252:37817 shutdown complete.
I20260620 14:48:28.565661 15757 master_runner.cc:305] Clearing existing system tablet
I20260620 14:48:28.566057 15757 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:28.566123 15757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:28.567381 15754 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98: attempting to promote NON_VOTER 3e72bddb92f142bb9e99468a591005e0 to VOTER
I20260620 14:48:28.567829 15754 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: NON_VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: true } }
I20260620 14:48:28.569195 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.568837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37817 (local address 127.0.0.1:45996)
0620 14:48:28.569018 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.569101 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 14:48:28.569396 15803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 1 LEARNER]: Refusing update from remote peer d3c4ab27b63c43d1b12d4e179d350d98: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 14:48:28.569921 15754 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 14:48:28.570300 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.569698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37817 (local address 127.0.0.1:46000)
0620 14:48:28.569867 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.569915 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 14:48:28.570428 15178 consensus_peers.cc:487] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 -> Peer 914c6ea4107a41b18d22d611649e61e2 (127.8.221.252:37817): Couldn't send request to peer 914c6ea4107a41b18d22d611649e61e2. Status: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 14:48:28.571916 15754 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3e72bddb92f142bb9e99468a591005e0 (127.8.221.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: NON_VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: true } } }
I20260620 14:48:28.573947 15753 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: NON_VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: true } } }
I20260620 14:48:28.574108 15753 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:28.573844 15943 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: NON_VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: true } } }
I20260620 14:48:28.574254 15943 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:28.574396 15803 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3e72bddb92f142bb9e99468a591005e0 (127.8.221.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: NON_VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: true } } }
I20260620 14:48:28.576313 15941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: NON_VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: true } } }
I20260620 14:48:28.576514 15941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:48:28.577044 15757 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.001s sys 0.000s
I20260620 14:48:28.578147 15944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:28.578387 15757 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:28.578518 15757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:28.578553 15757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/wal
uuid: "914c6ea4107a41b18d22d611649e61e2"
format_stamp: "Formatted at 2026-06-20 14:48:28 on dist-test-slave-65tz"
server_key: "cb731df68d1078d344bf94bee570526f"
server_key_iv: "75657f5ec7c876db6f4cac7a58fb9021"
server_key_version: "encryptionkey@0"
I20260620 14:48:28.580667 15757 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:28.583199 15757 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:48:28.583292 15757 log.cc:1196] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 14:48:28.583544 15757 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: Deleting consensus metadata
I20260620 14:48:28.583711 15757 master_runner.cc:315] Copying system tablet from 127.8.221.254:41225
I20260620 14:48:28.584898 15517 heartbeater.cc:503] Master 127.8.221.254:41225 was elected leader, sending a full tablet report...
I20260620 14:48:28.585690 15757 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: tablet copy: Beginning tablet copy session from remote peer at address 127.8.221.254:41225
I20260620 14:48:28.599334 15950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.586522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:53118)
0620 14:48:28.589987 (+ 3465us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.590003 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.590162 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.590486 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.590489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.590723 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.590876 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.590883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.592010 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.592013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.593068 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.593079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.593694 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.593698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.593823 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.594503 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.594517 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.594977 (+ 460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.597286 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.597290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.597292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.597550 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.597943 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.597947 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.597950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.598058 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.598661 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.598664 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.598785 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.598911 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.599087 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3386,"thread_start_us":99,"threads_started":1}
I20260620 14:48:28.599360 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.586246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53118 (local address 127.8.221.254:41225)
0620 14:48:28.586393 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.586399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.590122 (+ 3723us) server_negotiation.cc:407] Connection header received
0620 14:48:28.590280 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.590283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.590346 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.590446 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.591004 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.591889 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.593192 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.593569 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.593688 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.595137 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.595141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.595145 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.595171 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.597157 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.597662 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.597666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.597668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.597757 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.598183 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.598188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.598191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.598513 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.598635 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.599045 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.599195 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:48:28.600466 15212 tablet_copy_service.cc:140] P d3c4ab27b63c43d1b12d4e179d350d98: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 914c6ea4107a41b18d22d611649e61e2 ({username='oryx', principal='oryx/127.8.221.252@KRBTEST.COM'} at 127.0.0.1:53118)
I20260620 14:48:28.600585 15212 tablet_copy_service.cc:161] P d3c4ab27b63c43d1b12d4e179d350d98: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 914c6ea4107a41b18d22d611649e61e2 at {username='oryx', principal='oryx/127.8.221.252@KRBTEST.COM'} at 127.0.0.1:53118: session id = 914c6ea4107a41b18d22d611649e61e2-00000000000000000000000000000000
I20260620 14:48:28.601989 15212 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:48:28.603356 15757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:48:28.605759 15757 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: tablet copy: Starting download of 0 data blocks...
I20260620 14:48:28.605949 15757 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: tablet copy: Starting download of 1 WAL segments...
I20260620 14:48:28.607775 15757 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:48:28.609578 15212 tablet_copy_service.cc:342] P d3c4ab27b63c43d1b12d4e179d350d98: Request end of tablet copy session 914c6ea4107a41b18d22d611649e61e2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.221.252@KRBTEST.COM'} at 127.0.0.1:53118
I20260620 14:48:28.609666 15212 tablet_copy_service.cc:433] P d3c4ab27b63c43d1b12d4e179d350d98: ending tablet copy session 914c6ea4107a41b18d22d611649e61e2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 914c6ea4107a41b18d22d611649e61e2
I20260620 14:48:28.617565 15757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:28.619581 15951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:28.623003 15952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:28.627310 15757 server_base.cc:1034] running on GCE node
W20260620 14:48:28.627290 15954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:28.627661 15757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:28.628024 15757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:28.629189 15757 hybrid_clock.cc:648] HybridClock initialized: now 1781966908629164 us; error 51 us; skew 500 ppm
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966908, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:28.631076 15757 init.cc:377] Logged in from keytab as oryx/127.8.221.252@KRBTEST.COM (short username oryx)
I20260620 14:48:28.631673 15757 webserver.cc:460] Webserver started at http://127.8.221.252:43503/ using document root <none> and password file <none>
I20260620 14:48:28.631916 15757 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:28.631963 15757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:28.633554 15757 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:48:28.634927 15961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:28.635308 15757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:48:28.635398 15757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/wal
uuid: "914c6ea4107a41b18d22d611649e61e2"
format_stamp: "Formatted at 2026-06-20 14:48:28 on dist-test-slave-65tz"
server_key: "cb731df68d1078d344bf94bee570526f"
server_key_iv: "75657f5ec7c876db6f4cac7a58fb9021"
server_key_version: "encryptionkey@0"
I20260620 14:48:28.635525 15757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260620 14:48:28.661947 15937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:28.662240 15937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:28.662326 15937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:48:28.664570 15757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:28.667145 15757 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260620 14:48:28.667318 15937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:28.667402 15937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:28.667503 15937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:48:28.672659 15757 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.252:37817
I20260620 14:48:28.673430 15757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/master-2/data/info.pb
I20260620 14:48:28.675062 15937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:37325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.193.keytab
--principal=oryx/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:41493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=36711
--tserver_master_addrs=127.8.221.254:41225,127.8.221.253:37131,127.8.221.252:37817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:28.675485 16017 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:48:28.676466 15937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:28.676628 16017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: Bootstrap starting.
I20260620 14:48:28.677842 15937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:48:28.678189 16016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.252:37817 every 8 connection(s)
I20260620 14:48:28.688120 16017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:48:28.688470 16017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: Bootstrap complete.
I20260620 14:48:28.689043 16017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: NON_VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: true } }
I20260620 14:48:28.689330 16017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 914c6ea4107a41b18d22d611649e61e2, State: Initialized, Role: LEARNER
I20260620 14:48:28.689468 16017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: NON_VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: true } }
I20260620 14:48:28.689990 16017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:28.689947 16025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: NON_VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: true } } }
I20260620 14:48:28.690148 16025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [sys.catalog]: This master's current role is: LEARNER
W20260620 14:48:28.690820 16021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:28.690821 16022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:28.691507 16024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:28.692035 15937 server_base.cc:1034] running on GCE node
I20260620 14:48:28.692610 15937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:28.693369 16037 catalog_manager.cc:1259] Loaded cluster ID: 1337fddfcbd84c00afdea2e078edfce1
I20260620 14:48:28.693379 15937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:28.693492 16037 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: loading cluster ID for follower catalog manager: success
I20260620 14:48:28.695899 16037 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: acquiring CA information for follower catalog manager: success
I20260620 14:48:28.696681 15937 hybrid_clock.cc:648] HybridClock initialized: now 1781966908696640 us; error 58 us; skew 500 ppm
I20260620 14:48:28.698447 16037 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966908, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:28.700255 15937 init.cc:377] Logged in from keytab as oryx/127.8.221.193@KRBTEST.COM (short username oryx)
I20260620 14:48:28.701532 15937 webserver.cc:460] Webserver started at http://127.8.221.193:36711/ using document root <none> and password file <none>
I20260620 14:48:28.702347 15937 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:28.702461 15937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:28.707186 15937 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:48:28.709537 16043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:28.711104 15937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:28.711225 15937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "6b5d90216c0747ecbf9f808e728294ac"
format_stamp: "Formatted at 2026-06-20 14:48:27 on dist-test-slave-65tz"
server_key: "3060f6fc2e739f3f541fea2ff90e2a36"
server_key_iv: "22c087036d38a811678e4838b318109d"
server_key_version: "encryptionkey@0"
I20260620 14:48:28.711829 15937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:28.736865 15937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:28.740935 15937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:28.741149 15937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:28.741925 15937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:28.743049 15937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:28.743108 15937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:28.743176 15937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:28.743206 15937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:28.756114 15937 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:41493
I20260620 14:48:28.756138 16156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:41493 every 8 connection(s)
I20260620 14:48:28.757444 15937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:28.764257 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 15937
I20260620 14:48:28.764470 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 15388
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966908, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.253@KRBTEST.COM
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966908, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966908, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.252@KRBTEST.COM
I20260620 14:48:28.774264 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:38715
--local_ip_for_outbound_sockets=127.8.221.194
--tserver_master_addrs=127.8.221.254:41225
--webserver_port=43589
--webserver_interface=127.8.221.194
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.194.keytab
--principal=oryx/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:37325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.221.254:41225,127.8.221.253:37131
--tserver_master_addrs=127.8.221.254:41225,127.8.221.253:37131,127.8.221.252:37817 with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:48:28.779232 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.759780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:54285 (local address 127.8.221.253:37131)
0620 14:48:28.759965 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.759971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.761233 (+ 1262us) server_negotiation.cc:407] Connection header received
0620 14:48:28.762578 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.762581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.762654 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.762734 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.764532 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.765412 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.767749 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.768188 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.768281 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.773928 (+ 5647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.773932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.773935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.773963 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.776475 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.777131 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.777137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.777139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.777233 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.777557 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.777561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.777562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.777869 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.777955 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.778869 (+ 914us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.779016 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 14:48:28.779233 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.759745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:39031 (local address 127.8.221.252:37817)
0620 14:48:28.760166 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.760182 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.761214 (+ 1032us) server_negotiation.cc:407] Connection header received
0620 14:48:28.762610 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.762615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.762678 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.762753 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.764481 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.765740 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.767789 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.768197 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.768275 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.773983 (+ 5708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.773988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.773991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.774023 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.776126 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.776972 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.776977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.776979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.777077 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.777545 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.777548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.777550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.777844 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.777947 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.778840 (+ 893us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.779019 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":120,"threads_started":1}
I20260620 14:48:28.781399 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.760415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:37131 (local address 127.8.221.193:54285)
0620 14:48:28.761034 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.761093 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.762354 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.762914 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.762922 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.763435 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.764279 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.764302 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.765645 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.765649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.767601 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.767615 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.768500 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.768506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.768883 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.770064 (+ 1181us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.770090 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.773338 (+ 3248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.776618 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.776626 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.776630 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.776995 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.777357 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.777359 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.777361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.777444 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.778033 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.778042 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.778499 (+ 457us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.779932 (+ 1433us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.780067 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":474,"spinlock_wait_cycles":73216,"thread_start_us":196,"threads_started":1}
I20260620 14:48:28.788197 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.761893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:58563 (local address 127.8.221.254:41225)
0620 14:48:28.762056 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.762062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.762133 (+ 71us) server_negotiation.cc:407] Connection header received
0620 14:48:28.762610 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.762615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.762678 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.762749 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.764482 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.765445 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.767789 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.768186 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.768275 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.773306 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.773313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.773320 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.773365 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.785931 (+ 12566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.786656 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.786663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.786668 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.786767 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.787131 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.787137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.787138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.787429 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.787524 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.787889 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.788021 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:48:28.788781 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.761672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.193:58563)
0620 14:48:28.762045 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.762060 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.762375 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.762939 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.762943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.763488 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.764287 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.764306 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.765640 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.765643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.767635 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.767647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.768500 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.768504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.768869 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.769981 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.770012 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.773048 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.786142 (+ 13094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.786146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.786150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.786502 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.786900 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.786905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.786910 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.787011 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.787582 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.787585 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.787705 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.788400 (+ 695us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.788576 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":624,"spinlock_wait_cycles":26368,"thread_start_us":83,"threads_started":1}
I20260620 14:48:28.790529 16158 heartbeater.cc:347] Connected to a master server at 127.8.221.254:41225
I20260620 14:48:28.790874 16158 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:28.791344 16159 heartbeater.cc:347] Connected to a master server at 127.8.221.253:37131
I20260620 14:48:28.791430 16159 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:28.791657 16158 heartbeater.cc:511] Master 127.8.221.254:41225 requested a full tablet report, sending...
I20260620 14:48:28.791657 16159 heartbeater.cc:511] Master 127.8.221.253:37131 requested a full tablet report, sending...
I20260620 14:48:28.781416 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.760411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37817 (local address 127.8.221.193:39031)
0620 14:48:28.761031 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.761094 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.762376 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.762939 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.762943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.763488 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.764288 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.764345 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.765932 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.765936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.767635 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.767647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.768500 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.768504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.768888 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.770058 (+ 1170us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.770080 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.773763 (+ 3683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.776288 (+ 2525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.776296 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.776334 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.776838 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.777201 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.777204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.777206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.777422 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.778033 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.778042 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.778499 (+ 457us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.779530 (+ 1031us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.779983 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":566,"spinlock_wait_cycles":37248,"thread_start_us":147,"threads_started":1}
I20260620 14:48:28.793359 15192 ts_manager.cc:194] Re-registered known tserver with Master: 6b5d90216c0747ecbf9f808e728294ac (127.8.221.193:41493)
I20260620 14:48:28.793737 15793 ts_manager.cc:194] Registered new tserver with Master: 6b5d90216c0747ecbf9f808e728294ac (127.8.221.193:41493)
I20260620 14:48:28.794341 15192 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:58563
I20260620 14:48:28.795902 16157 heartbeater.cc:347] Connected to a master server at 127.8.221.252:37817
I20260620 14:48:28.796017 16157 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:28.796214 16157 heartbeater.cc:511] Master 127.8.221.252:37817 requested a full tablet report, sending...
I20260620 14:48:28.797523 15973 ts_manager.cc:194] Registered new tserver with Master: 6b5d90216c0747ecbf9f808e728294ac (127.8.221.193:41493)
I20260620 14:48:28.823261 15651 heartbeater.cc:503] Master 127.8.221.254:41225 was elected leader, sending a full tablet report...
W20260620 14:48:28.897110 16165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:28.897400 16165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:28.897477 16165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:28.901832 16165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:28.901914 16165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:28.902007 16165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:48:28.907271 16165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:37325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.194.keytab
--principal=oryx/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:38715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=43589
--tserver_master_addrs=127.8.221.254:41225,127.8.221.253:37131,127.8.221.252:37817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:28.908329 16165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:28.909349 16165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:28.919663 16173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:28.919732 16170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:28.919670 16171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:28.920138 16165 server_base.cc:1034] running on GCE node
I20260620 14:48:28.920534 16165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:28.921212 16165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:28.922422 16165 hybrid_clock.cc:648] HybridClock initialized: now 1781966908922380 us; error 61 us; skew 500 ppm
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966908, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:28.925447 16165 init.cc:377] Logged in from keytab as oryx/127.8.221.194@KRBTEST.COM (short username oryx)
I20260620 14:48:28.926486 16165 webserver.cc:460] Webserver started at http://127.8.221.194:43589/ using document root <none> and password file <none>
I20260620 14:48:28.927129 16165 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:28.927241 16165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:28.931706 16165 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:48:28.934082 16180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:28.935734 16165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:28.935886 16165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b"
format_stamp: "Formatted at 2026-06-20 14:48:27 on dist-test-slave-65tz"
server_key: "961c730d0ee3a151665f7c27f11cd4e7"
server_key_iv: "0a9f076f3c69314b2fc2883b0da225ae"
server_key_version: "encryptionkey@0"
I20260620 14:48:28.936617 16165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:28.954555 16165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:28.957921 16165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:28.958129 16165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:28.958827 16165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:28.959815 16165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:28.959877 16165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:28.959940 16165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:28.959975 16165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:28.970863 16165 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:38715
I20260620 14:48:28.971307 16293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:38715 every 8 connection(s)
I20260620 14:48:28.971935 16165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:28.972301 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 16165
I20260620 14:48:28.972537 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 15522
I20260620 14:48:28.983682 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:34701
--local_ip_for_outbound_sockets=127.8.221.195
--tserver_master_addrs=127.8.221.254:41225
--webserver_port=45893
--webserver_interface=127.8.221.195
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.195.keytab
--principal=oryx/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:37325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.221.254:41225,127.8.221.253:37131
--tserver_master_addrs=127.8.221.254:41225,127.8.221.253:37131,127.8.221.252:37817 with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966908, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.252@KRBTEST.COM
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966908, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM
Jun 20 14:48:28 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966908, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.253@KRBTEST.COM
I20260620 14:48:28.992959 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.975632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:42467 (local address 127.8.221.254:41225)
0620 14:48:28.975788 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.975795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.975929 (+ 134us) server_negotiation.cc:407] Connection header received
0620 14:48:28.977259 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.977262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.977313 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.977368 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.979651 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.980512 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.982665 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.982997 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.983075 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.987655 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.987658 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.987661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.987692 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.990390 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.990996 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.991008 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.991012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.991089 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.991520 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.991523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.991525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.991742 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.991823 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.992666 (+ 843us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.992788 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:48:28.992959 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.976460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:33543 (local address 127.8.221.253:37131)
0620 14:48:28.976568 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.976573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.976586 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:48:28.977258 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.977261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.977313 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.977368 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.979672 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.980522 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.982665 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.982997 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.983072 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.988122 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.988126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.988129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.988162 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.990132 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.990973 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.990980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.990981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.991065 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.991526 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.991529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.991532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.991742 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.991823 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.992666 (+ 843us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.992788 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:48:28.993180 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.974295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:57657 (local address 127.8.221.252:37817)
0620 14:48:28.974503 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.974510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.975235 (+ 725us) server_negotiation.cc:407] Connection header received
0620 14:48:28.977295 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.977302 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.977373 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.977438 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.979658 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.980515 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.982656 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.983057 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.983150 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.988250 (+ 5100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:28.988256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:28.988263 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:28.988310 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:28.990367 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.991170 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.991177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.991181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.991290 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:28.991641 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:28.991647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:28.991649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:28.991966 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:28.992069 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:28.992732 (+ 663us) server_negotiation.cc:299] Negotiation successful
0620 14:48:28.992915 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 14:48:28.994801 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.975532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.194:42467)
0620 14:48:28.975823 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.975838 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.976995 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.977818 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.977828 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.978555 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.979466 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.979488 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.980699 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.980703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.982525 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.982535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.983277 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.983281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.983528 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.984432 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.984463 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.987469 (+ 3006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.990510 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.990513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.990514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.990890 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.991189 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.991193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.991194 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.991405 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.991893 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.991901 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.992349 (+ 448us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.993698 (+ 1349us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.993870 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":540,"spinlock_wait_cycles":45184,"thread_start_us":104,"threads_started":1}
I20260620 14:48:28.994807 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.976053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:37131 (local address 127.8.221.194:33543)
0620 14:48:28.976331 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.976345 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.976995 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.977827 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.977830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.978549 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.979472 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.979488 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.980728 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.980731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.982525 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.982535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.983277 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.983281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.983526 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.984632 (+ 1106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.984649 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.987962 (+ 3313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.990312 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.990319 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.990339 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.990853 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.991182 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.991185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.991187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.991405 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.991916 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.991919 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.992349 (+ 430us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.993207 (+ 858us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.993600 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":670,"spinlock_wait_cycles":69632,"thread_start_us":147,"threads_started":1}
I20260620 14:48:28.994808 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.974386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37817 (local address 127.8.221.194:57657)
0620 14:48:28.975022 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.975092 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.976984 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.977825 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.977832 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.978555 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.979467 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.979489 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.980704 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.980709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.982512 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.982526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.983438 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.983443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.983633 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.984526 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.984548 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.988068 (+ 3520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:28.990567 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.990574 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.990579 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.991038 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.991415 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:28.991421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:28.991423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:28.991524 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:28.992162 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:28.992168 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:28.992351 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 14:48:28.993522 (+ 1171us) client_negotiation.cc:240] Negotiation successful
0620 14:48:28.993731 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":335,"spinlock_wait_cycles":15872,"thread_start_us":128,"threads_started":1}
I20260620 14:48:28.996343 16295 heartbeater.cc:347] Connected to a master server at 127.8.221.254:41225
I20260620 14:48:28.996372 16294 heartbeater.cc:347] Connected to a master server at 127.8.221.252:37817
I20260620 14:48:28.996594 16295 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:28.996594 16294 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:28.997107 16296 heartbeater.cc:347] Connected to a master server at 127.8.221.253:37131
I20260620 14:48:28.997190 16296 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:28.997267 16294 heartbeater.cc:511] Master 127.8.221.252:37817 requested a full tablet report, sending...
I20260620 14:48:28.997334 16296 heartbeater.cc:511] Master 127.8.221.253:37131 requested a full tablet report, sending...
I20260620 14:48:28.998121 16295 heartbeater.cc:511] Master 127.8.221.254:41225 requested a full tablet report, sending...
I20260620 14:48:28.998584 15973 ts_manager.cc:194] Registered new tserver with Master: d3c00e694fc84e3eaf83fd9ec6a87d0b (127.8.221.194:38715)
I20260620 14:48:28.998697 15793 ts_manager.cc:194] Registered new tserver with Master: d3c00e694fc84e3eaf83fd9ec6a87d0b (127.8.221.194:38715)
I20260620 14:48:28.998795 15192 ts_manager.cc:194] Re-registered known tserver with Master: d3c00e694fc84e3eaf83fd9ec6a87d0b (127.8.221.194:38715)
I20260620 14:48:28.999557 15192 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:42467
I20260620 14:48:29.011237 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.007843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37817 (local address 127.0.0.1:46016)
0620 14:48:29.007956 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:29.007971 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:29.008077 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:29.008351 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:29.008353 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:29.008368 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:29.008463 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.008468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.009225 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.009231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.010157 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.010167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.010844 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.010847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.010928 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.010948 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:48:29.011013 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:48:29.011102 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 14:48:29.011237 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.007957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46016 (local address 127.8.221.252:37817)
0620 14:48:29.008110 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.008116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.008133 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:48:29.008185 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.008188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.008245 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.008343 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:29.008574 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.009117 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.010271 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.010707 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.010840 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.011010 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:29.011045 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 14:48:29.011102 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:48:29.012236 15984 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 14:48:29.012674 16025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c4ab27b63c43d1b12d4e179d350d98. Latest consensus state: current_term: 1 leader_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: NON_VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: true } } }
I20260620 14:48:29.012862 16025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [sys.catalog]: This master's current role is: LEARNER
W20260620 14:48:29.116777 16301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:29.117122 16301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:29.117205 16301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:29.121398 16301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:29.121506 16301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:29.121618 16301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:48:29.127213 16301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:37325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.195.keytab
--principal=oryx/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:34701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=45893
--tserver_master_addrs=127.8.221.254:41225,127.8.221.253:37131,127.8.221.252:37817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:29.128425 16301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:29.129559 16301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:29.140964 16306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:29.140973 16309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:29.141467 16301 server_base.cc:1034] running on GCE node
W20260620 14:48:29.141475 16307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:29.142112 16301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:29.142947 16301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:29.144182 16301 hybrid_clock.cc:648] HybridClock initialized: now 1781966909144166 us; error 40 us; skew 500 ppm
Jun 20 14:48:29 dist-test-slave-65tz krb5kdc[15147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966909, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:29.147569 16301 init.cc:377] Logged in from keytab as oryx/127.8.221.195@KRBTEST.COM (short username oryx)
I20260620 14:48:29.148577 16301 webserver.cc:460] Webserver started at http://127.8.221.195:45893/ using document root <none> and password file <none>
I20260620 14:48:29.149190 16301 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:29.149302 16301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:29.153429 16301 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:29.155738 16316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:29.157125 16301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:29.157272 16301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "113fb3b62558486f8265f5bc4311d8a2"
format_stamp: "Formatted at 2026-06-20 14:48:27 on dist-test-slave-65tz"
server_key: "d161557c9aadc1b6a07c79fed97e19cd"
server_key_iv: "aa7d6e19d00f93d57fd47c2f2e9285c8"
server_key_version: "encryptionkey@0"
I20260620 14:48:29.157927 16301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:29.171746 16301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:29.175187 16301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:29.175405 16301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:29.176025 16301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:29.176968 16301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:29.177047 16301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:29.177126 16301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:29.177171 16301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:29.187815 16301 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:34701
I20260620 14:48:29.187904 16429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:34701 every 8 connection(s)
I20260620 14:48:29.189049 16301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:29.193734 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 16301
Jun 20 14:48:29 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966909, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM
Jun 20 14:48:29 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966909, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.253@KRBTEST.COM
Jun 20 14:48:29 dist-test-slave-65tz krb5kdc[15147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966909, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.252@KRBTEST.COM
I20260620 14:48:29.210345 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.191590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:56101 (local address 127.8.221.254:41225)
0620 14:48:29.191776 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.191781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.192919 (+ 1138us) server_negotiation.cc:407] Connection header received
0620 14:48:29.195380 (+ 2461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.195384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.195453 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.195593 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:29.197596 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.198522 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.200414 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.200760 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.200856 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.204807 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:29.204812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:29.204816 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:29.204847 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:29.207824 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.208453 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.208457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.208459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.208539 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.208922 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.208925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.208927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.209106 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:29.209196 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:29.210010 (+ 814us) server_negotiation.cc:299] Negotiation successful
0620 14:48:29.210152 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 14:48:29.211452 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.191390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:56491 (local address 127.8.221.253:37131)
0620 14:48:29.191609 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.191615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.192888 (+ 1273us) server_negotiation.cc:407] Connection header received
0620 14:48:29.194477 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.194483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.194554 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.194613 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:29.199274 (+ 4661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.200098 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.201246 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.201561 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.201648 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.205201 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:29.205208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:29.205227 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:29.205253 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:29.208658 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.209328 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.209333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.209335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.209438 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.210004 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.210008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.210010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.210286 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:29.210385 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:29.211100 (+ 715us) server_negotiation.cc:299] Negotiation successful
0620 14:48:29.211241 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 14:48:29.211887 16434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.192021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.195:56101)
0620 14:48:29.192690 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:29.192758 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:29.194191 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:29.195630 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:29.195632 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:29.196576 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:29.197393 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.197414 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.198715 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.198719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.200283 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.200290 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.201067 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.201071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.201306 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.202122 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:29.202145 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:29.204603 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:29.207954 (+ 3351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:29.207962 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:29.207980 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:29.208332 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:29.208632 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:29.208635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:29.208637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:29.208818 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:29.209254 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:29.209262 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:29.209641 (+ 379us) client_negotiation.cc:769] Sending connection context
0620 14:48:29.210157 (+ 516us) client_negotiation.cc:240] Negotiation successful
0620 14:48:29.210544 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":514,"spinlock_wait_cycles":51584,"thread_start_us":155,"threads_started":1}
I20260620 14:48:29.212677 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.192021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:37131 (local address 127.8.221.195:56491)
0620 14:48:29.192690 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:29.192758 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:29.194191 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:29.195098 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:29.195108 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:29.196483 (+ 1375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:29.199124 (+ 2641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.199134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.200283 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.200286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.201100 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.201108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.201834 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.201839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.201933 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.202688 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:29.202703 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:29.205028 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:29.208819 (+ 3791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:29.208826 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:29.208829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:29.209174 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:29.209594 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:29.209599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:29.209603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:29.209715 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:29.210735 (+ 1020us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:29.210738 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:29.210879 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 14:48:29.211852 (+ 973us) client_negotiation.cc:240] Negotiation successful
0620 14:48:29.212383 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":1911,"spinlock_wait_cycles":27520,"thread_start_us":169,"threads_started":1}
I20260620 14:48:29.214020 16431 heartbeater.cc:347] Connected to a master server at 127.8.221.254:41225
I20260620 14:48:29.214057 16432 heartbeater.cc:347] Connected to a master server at 127.8.221.253:37131
I20260620 14:48:29.214138 16446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.196835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.8.221.254:41225)
0620 14:48:29.197130 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.197143 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.197159 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:48:29.197200 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.197203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.197250 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.197354 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:29.198633 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.199591 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.201105 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.201479 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.201585 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.203850 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:29.203854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:29.203857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:29.203886 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:29.206509 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.207090 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.207096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.207098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.207216 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.207590 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.207593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.207595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.207856 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:29.213698 (+ 5842us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:29.213783 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 14:48:29.213961 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":103,"threads_started":1}
I20260620 14:48:29.214308 16432 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:29.215026 16432 heartbeater.cc:511] Master 127.8.221.253:37131 requested a full tablet report, sending...
I20260620 14:48:29.214308 16431 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:29.215925 16431 heartbeater.cc:511] Master 127.8.221.254:41225 requested a full tablet report, sending...
I20260620 14:48:29.216902 15793 ts_manager.cc:194] Registered new tserver with Master: 113fb3b62558486f8265f5bc4311d8a2 (127.8.221.195:34701)
I20260620 14:48:29.217006 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.193971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:59611 (local address 127.8.221.252:37817)
0620 14:48:29.194127 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.194133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.194158 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:48:29.194553 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.194557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.194613 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.194664 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:29.197649 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.198522 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.203570 (+ 5048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.203905 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.203999 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.212731 (+ 8732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:29.212735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:29.212737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:29.212764 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:29.214527 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.215347 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.215353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.215355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.215464 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.215889 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.215893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.215895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.216134 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:29.216237 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:29.216683 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 14:48:29.216833 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:48:29.217598 16436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.193293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37817 (local address 127.8.221.195:59611)
0620 14:48:29.193590 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:29.193606 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:29.194418 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:29.195098 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:29.195108 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:29.197369 (+ 2261us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:29.197506 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.197515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.198715 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.198719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.203427 (+ 4708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.203440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.204845 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.204848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.204952 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.206001 (+ 1049us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:29.206016 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:29.212569 (+ 6553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:29.214891 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:29.214895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:29.214897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:29.215215 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:29.215579 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:29.215583 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:29.215585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:29.215715 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:29.216274 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:29.216277 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:29.216382 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 14:48:29.217182 (+ 800us) client_negotiation.cc:240] Negotiation successful
0620 14:48:29.217352 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":35,"thread_start_us":92,"threads_started":1}
I20260620 14:48:29.218181 15192 ts_manager.cc:194] Re-registered known tserver with Master: 113fb3b62558486f8265f5bc4311d8a2 (127.8.221.195:34701)
I20260620 14:48:29.218451 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.196181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45444 (local address 127.8.221.253:37131)
0620 14:48:29.197081 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.197086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.202443 (+ 5357us) server_negotiation.cc:407] Connection header received
0620 14:48:29.205235 (+ 2792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.205238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.205294 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.205368 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:29.206370 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.207042 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.208819 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.209182 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.209265 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.211542 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:29.211546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:29.211549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:29.211572 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:29.215399 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.216966 (+ 1567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.216969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.216971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.217058 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.217345 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.217349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.217350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.217581 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:29.217703 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:29.218173 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 14:48:29.218302 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":69,"threads_started":1}
I20260620 14:48:29.218720 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.196734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46020 (local address 127.8.221.252:37817)
0620 14:48:29.197562 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.197568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.197583 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:48:29.197633 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.197636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.197705 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.198718 (+ 1013us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:29.198828 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.199611 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.201105 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.201466 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.204157 (+ 2691us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.204304 (+ 147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:29.204309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:29.204313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:29.204338 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:29.210723 (+ 6385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.211287 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.211290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.211292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.211376 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.211727 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.211731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.211733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.212070 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:29.218345 (+ 6275us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:29.218429 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 14:48:29.218564 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":92,"threads_started":1}
I20260620 14:48:29.218921 15192 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:56101
I20260620 14:48:29.218557 15191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:48:29.222184 15191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:29.223275 16430 heartbeater.cc:347] Connected to a master server at 127.8.221.252:37817
I20260620 14:48:29.223384 16430 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:29.223541 16430 heartbeater.cc:511] Master 127.8.221.252:37817 requested a full tablet report, sending...
I20260620 14:48:29.224396 15973 ts_manager.cc:194] Registered new tserver with Master: 113fb3b62558486f8265f5bc4311d8a2 (127.8.221.195:34701)
I20260620 14:48:29.225672 15939 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: NON_VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: true } } }
I20260620 14:48:29.225948 15939 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:29.229970 16448 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98: attempting to promote NON_VOTER 914c6ea4107a41b18d22d611649e61e2 to VOTER
I20260620 14:48:29.230335 16448 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } }
I20260620 14:48:29.231492 15803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 1 FOLLOWER]: Refusing update from remote peer d3c4ab27b63c43d1b12d4e179d350d98: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 14:48:29.231866 16449 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 14:48:29.232504 15984 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 1 LEARNER]: Refusing update from remote peer d3c4ab27b63c43d1b12d4e179d350d98: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 14:48:29.232911 16449 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 14:48:29.234404 16448 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 914c6ea4107a41b18d22d611649e61e2 (127.8.221.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } } }
I20260620 14:48:29.235004 15803 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 914c6ea4107a41b18d22d611649e61e2 (127.8.221.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } } }
I20260620 14:48:29.235735 15984 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 914c6ea4107a41b18d22d611649e61e2 (127.8.221.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } } }
I20260620 14:48:29.236032 16448 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } } }
I20260620 14:48:29.236123 16448 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:29.236258 16448 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } } }
I20260620 14:48:29.236361 16448 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:29.237226 16450 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } } }
I20260620 14:48:29.237355 16450 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:29.237344 16458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } } }
I20260620 14:48:29.237475 16458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:48:29.238991 16025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } } }
I20260620 14:48:29.239101 16025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:48:29.246836 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.240603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:34701 (local address 127.0.0.1:48422)
0620 14:48:29.241172 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:29.241189 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:29.241302 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:29.242963 (+ 1661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:29.242966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:29.242986 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:29.243081 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.243087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.244365 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.244370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.245229 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.245237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.246355 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.246358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.246481 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.246535 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 14:48:29.246611 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:48:29.246689 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":28,"thread_start_us":92,"threads_started":1}
I20260620 14:48:29.247293 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.241265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48422 (local address 127.8.221.195:34701)
0620 14:48:29.242531 (+ 1266us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.242544 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.242566 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:48:29.242633 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.242637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.242808 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.242929 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:29.243215 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.244263 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.245427 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.246255 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.246356 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.246500 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:29.246640 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 14:48:29.246759 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1156,"thread_start_us":126,"threads_started":1}
I20260620 14:48:29.247330 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.240007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:38715 (local address 127.0.0.1:35782)
0620 14:48:29.240157 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:29.240176 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:29.240264 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:29.242339 (+ 2075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:29.242343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:29.242360 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:29.242497 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.242507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.243878 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.243884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.245415 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.245427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.246781 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.246788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.246920 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.246965 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:48:29.247030 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:48:29.247136 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":95}
I20260620 14:48:29.248570 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.240150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35782 (local address 127.8.221.194:38715)
0620 14:48:29.240669 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.240685 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.240712 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:48:29.241865 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.241874 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.242115 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.242286 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:29.242675 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.243745 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.245561 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.246601 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.248064 (+ 1463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.248241 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:29.248300 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 14:48:29.248402 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":100,"threads_started":1}
I20260620 14:48:29.250025 16364 tablet_service.cc:1467] Processing CreateTablet for tablet fa92ab8198224f0cb8cf558b679aa630 (DEFAULT_TABLE table=test-table [id=5820d30866794adbadf923b6c6c76358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:29.250583 16460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.240914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:41493 (local address 127.0.0.1:53902)
0620 14:48:29.243118 (+ 2204us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:29.243131 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:29.243243 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:29.243840 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:29.243843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:29.243860 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:29.244002 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.244008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.245554 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.245558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.248735 (+ 3177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.248746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.250131 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.250135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.250229 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.250269 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:48:29.250348 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:48:29.250430 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2144,"mutex_wait_us":3,"thread_start_us":128,"threads_started":1}
I20260620 14:48:29.251019 16364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa92ab8198224f0cb8cf558b679aa630. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:29.251070 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.241415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53902 (local address 127.8.221.193:41493)
0620 14:48:29.241746 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.241761 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.243340 (+ 1579us) server_negotiation.cc:407] Connection header received
0620 14:48:29.243412 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.243417 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.243590 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.243720 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:29.244141 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.245394 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.248890 (+ 3496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.249776 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.249913 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.250060 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:29.250782 (+ 722us) server_negotiation.cc:299] Negotiation successful
0620 14:48:29.250905 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":82,"threads_started":1}
I20260620 14:48:29.251852 16228 tablet_service.cc:1467] Processing CreateTablet for tablet fa92ab8198224f0cb8cf558b679aa630 (DEFAULT_TABLE table=test-table [id=5820d30866794adbadf923b6c6c76358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:29.252892 16228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa92ab8198224f0cb8cf558b679aa630. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:29.257620 16464 tablet_bootstrap.cc:492] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2: Bootstrap starting.
I20260620 14:48:29.260326 16464 tablet_bootstrap.cc:654] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:29.260821 16091 tablet_service.cc:1467] Processing CreateTablet for tablet fa92ab8198224f0cb8cf558b679aa630 (DEFAULT_TABLE table=test-table [id=5820d30866794adbadf923b6c6c76358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:29.261286 16464 log.cc:824] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:29.261886 16091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa92ab8198224f0cb8cf558b679aa630. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:29.262084 16465 tablet_bootstrap.cc:492] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b: Bootstrap starting.
I20260620 14:48:29.263926 16464 tablet_bootstrap.cc:492] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2: No bootstrap required, opened a new log
I20260620 14:48:29.264180 16464 ts_tablet_manager.cc:1397] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 14:48:29.264549 16465 tablet_bootstrap.cc:654] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:29.265293 16465 log.cc:824] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:29.267504 16465 tablet_bootstrap.cc:492] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b: No bootstrap required, opened a new log
I20260620 14:48:29.267696 16465 ts_tablet_manager.cc:1397] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:48:29.268489 16464 raft_consensus.cc:348] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } } peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } }
I20260620 14:48:29.268790 16464 raft_consensus.cc:374] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:29.268850 16464 raft_consensus.cc:729] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 113fb3b62558486f8265f5bc4311d8a2, State: Initialized, Role: FOLLOWER
I20260620 14:48:29.269371 16464 consensus_queue.cc:260] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } } peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } }
I20260620 14:48:29.270622 16464 ts_tablet_manager.cc:1428] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:48:29.271308 16431 heartbeater.cc:503] Master 127.8.221.254:41225 was elected leader, sending a full tablet report...
I20260620 14:48:29.272259 16465 raft_consensus.cc:348] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } } peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } }
I20260620 14:48:29.272526 16465 raft_consensus.cc:374] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:29.272575 16465 raft_consensus.cc:729] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c00e694fc84e3eaf83fd9ec6a87d0b, State: Initialized, Role: FOLLOWER
I20260620 14:48:29.273036 16465 consensus_queue.cc:260] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } } peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } }
I20260620 14:48:29.274078 16465 ts_tablet_manager.cc:1428] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:48:29.274284 16295 heartbeater.cc:503] Master 127.8.221.254:41225 was elected leader, sending a full tablet report...
I20260620 14:48:29.277213 16469 tablet_bootstrap.cc:492] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac: Bootstrap starting.
I20260620 14:48:29.279042 16469 tablet_bootstrap.cc:654] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:29.279788 16469 log.cc:824] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:29.281773 16469 tablet_bootstrap.cc:492] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac: No bootstrap required, opened a new log
I20260620 14:48:29.282054 16469 ts_tablet_manager.cc:1397] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260620 14:48:29.284860 16469 raft_consensus.cc:348] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } } peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } }
I20260620 14:48:29.285069 16469 raft_consensus.cc:374] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:29.285104 16469 raft_consensus.cc:729] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b5d90216c0747ecbf9f808e728294ac, State: Initialized, Role: FOLLOWER
I20260620 14:48:29.285579 16469 consensus_queue.cc:260] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } } peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } }
I20260620 14:48:29.286329 16158 heartbeater.cc:503] Master 127.8.221.254:41225 was elected leader, sending a full tablet report...
I20260620 14:48:29.286978 16469 ts_tablet_manager.cc:1428] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260620 14:48:29.409649 16470 raft_consensus.cc:482] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:29.409999 16470 raft_consensus.cc:504] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } } peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } }
I20260620 14:48:29.411561 16470 leader_election.cc:288] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b5d90216c0747ecbf9f808e728294ac (127.8.221.193:41493), 113fb3b62558486f8265f5bc4311d8a2 (127.8.221.195:34701)
I20260620 14:48:29.415756 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.412023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:34701 (local address 127.8.221.194:55199)
0620 14:48:29.412257 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:29.412275 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:29.412424 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:29.412703 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:29.412706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:29.412721 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:29.412863 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.412869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.413595 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.413599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.414618 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.414627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.415279 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.415282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.415378 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.415408 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 14:48:29.415507 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:48:29.415599 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":22}
I20260620 14:48:29.415792 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.412119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:55199 (local address 127.8.221.195:34701)
0620 14:48:29.412251 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.412256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.412359 (+ 103us) server_negotiation.cc:407] Connection header received
0620 14:48:29.412559 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.412567 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.412606 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.412701 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:29.412980 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.413481 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.414749 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.415149 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.415290 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.415344 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:29.415520 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 14:48:29.415590 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:48:29.415799 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.412006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:57041 (local address 127.8.221.193:41493)
0620 14:48:29.412160 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.412166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.412281 (+ 115us) server_negotiation.cc:407] Connection header received
0620 14:48:29.412476 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.412480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.412533 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.412621 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:29.412906 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.413464 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.414723 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.415181 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.415319 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.415384 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:29.415568 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 14:48:29.415644 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:48:29.415799 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.411909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:41493 (local address 127.8.221.194:57041)
0620 14:48:29.412165 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:29.412183 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:29.412353 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:29.412632 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:29.412635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:29.412650 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:29.412795 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.412801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.413580 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.413585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.414597 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.414606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.415313 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.415317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.415429 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.415459 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 14:48:29.415552 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:48:29.415638 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":80}
I20260620 14:48:29.416554 16111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa92ab8198224f0cb8cf558b679aa630" candidate_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b5d90216c0747ecbf9f808e728294ac" is_pre_election: true
I20260620 14:48:29.416953 16111 raft_consensus.cc:2454] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3c00e694fc84e3eaf83fd9ec6a87d0b in term 0.
I20260620 14:48:29.416993 16384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa92ab8198224f0cb8cf558b679aa630" candidate_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113fb3b62558486f8265f5bc4311d8a2" is_pre_election: true
I20260620 14:48:29.417346 16384 raft_consensus.cc:2454] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3c00e694fc84e3eaf83fd9ec6a87d0b in term 0.
I20260620 14:48:29.417600 16182 leader_election.cc:302] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b5d90216c0747ecbf9f808e728294ac, d3c00e694fc84e3eaf83fd9ec6a87d0b; no voters:
I20260620 14:48:29.418105 16470 raft_consensus.cc:2781] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:29.418193 16470 raft_consensus.cc:482] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:29.418226 16470 raft_consensus.cc:3037] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:29.419282 16470 raft_consensus.cc:504] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } } peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } }
I20260620 14:48:29.419699 16470 leader_election.cc:288] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [CANDIDATE]: Term 1 election: Requested vote from peers 6b5d90216c0747ecbf9f808e728294ac (127.8.221.193:41493), 113fb3b62558486f8265f5bc4311d8a2 (127.8.221.195:34701)
I20260620 14:48:29.420068 16384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa92ab8198224f0cb8cf558b679aa630" candidate_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113fb3b62558486f8265f5bc4311d8a2"
I20260620 14:48:29.420063 16111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa92ab8198224f0cb8cf558b679aa630" candidate_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b5d90216c0747ecbf9f808e728294ac"
I20260620 14:48:29.420169 16384 raft_consensus.cc:3037] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:29.420204 16111 raft_consensus.cc:3037] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:29.421180 16384 raft_consensus.cc:2454] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3c00e694fc84e3eaf83fd9ec6a87d0b in term 1.
I20260620 14:48:29.421464 16184 leader_election.cc:302] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 113fb3b62558486f8265f5bc4311d8a2, d3c00e694fc84e3eaf83fd9ec6a87d0b; no voters:
I20260620 14:48:29.421510 16111 raft_consensus.cc:2454] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3c00e694fc84e3eaf83fd9ec6a87d0b in term 1.
I20260620 14:48:29.421631 16470 raft_consensus.cc:2781] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:29.421921 16470 raft_consensus.cc:686] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 1 LEADER]: Becoming Leader. State: Replica: d3c00e694fc84e3eaf83fd9ec6a87d0b, State: Running, Role: LEADER
I20260620 14:48:29.422559 16470 consensus_queue.cc:237] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } } peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } }
I20260620 14:48:29.426241 15192 catalog_manager.cc:5495] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b reported cstate change: term changed from 0 to 1, leader changed from <none> to d3c00e694fc84e3eaf83fd9ec6a87d0b (127.8.221.194). New cstate: current_term: 1 leader_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } health_report { overall_health: UNKNOWN } } }
W20260620 14:48:29.440723 16433 tablet.cc:2369] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:48:29.474654 16297 tablet.cc:2369] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:48:29.484587 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.480459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35798 (local address 127.8.221.194:38715)
0620 14:48:29.480691 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.480703 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.480721 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:48:29.480831 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.480833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.480907 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.481034 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:29.481363 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.482012 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.482903 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.483114 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.483200 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.483476 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:29.484010 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:29.484119 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:29.484336 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 14:48:29.484431 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:48:29.492277 16111 raft_consensus.cc:1261] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac [term 1 FOLLOWER]: Refusing update from remote peer d3c00e694fc84e3eaf83fd9ec6a87d0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:29.492280 16384 raft_consensus.cc:1261] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2 [term 1 FOLLOWER]: Refusing update from remote peer d3c00e694fc84e3eaf83fd9ec6a87d0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:29.493266 16470 consensus_queue.cc:1035] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:29.493535 16473 consensus_queue.cc:1035] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:29.504130 16478 mvcc.cc:205] Tried to move back new op lower bound from 7298936461270605824 to 7298936460997713920. Current Snapshot: MvccSnapshot[applied={T|T < 7298936461270605824}]
I20260620 14:48:29.509757 16477 mvcc.cc:205] Tried to move back new op lower bound from 7298936461270605824 to 7298936460997713920. Current Snapshot: MvccSnapshot[applied={T|T < 7298936461270605824}]
I20260620 14:48:29.518874 15192 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:29.519048 15192 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:29.524765 15192 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3c4ab27b63c43d1b12d4e179d350d98: Sending DeleteTablet for 3 replicas of tablet fa92ab8198224f0cb8cf558b679aa630
I20260620 14:48:29.525614 16228 tablet_service.cc:1514] Processing DeleteTablet for tablet fa92ab8198224f0cb8cf558b679aa630 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:29 UTC) from {username='oryx', principal='oryx/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:35782
I20260620 14:48:29.525717 16364 tablet_service.cc:1514] Processing DeleteTablet for tablet fa92ab8198224f0cb8cf558b679aa630 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:29 UTC) from {username='oryx', principal='oryx/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:48422
I20260620 14:48:29.526224 16485 tablet_replica.cc:330] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b: stopping tablet replica
I20260620 14:48:29.526252 16091 tablet_service.cc:1514] Processing DeleteTablet for tablet fa92ab8198224f0cb8cf558b679aa630 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:29 UTC) from {username='oryx', principal='oryx/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:53902
I20260620 14:48:29.526762 16486 tablet_replica.cc:330] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2: stopping tablet replica
I20260620 14:48:29.526780 16487 tablet_replica.cc:330] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac: stopping tablet replica
I20260620 14:48:29.527137 16486 raft_consensus.cc:2229] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:29.527137 16487 raft_consensus.cc:2229] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:29.527443 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 15160
I20260620 14:48:29.527524 16486 raft_consensus.cc:2258] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:29.527508 16485 raft_consensus.cc:2229] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:29.528797 16485 raft_consensus.cc:2258] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:29.530134 16485 ts_tablet_manager.cc:1905] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:29.530134 16486 ts_tablet_manager.cc:1905] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:29.533221 16486 ts_tablet_manager.cc:1918] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:29.533213 16485 ts_tablet_manager.cc:1918] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:29.533322 16486 log.cc:1196] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/fa92ab8198224f0cb8cf558b679aa630
I20260620 14:48:29.533453 16485 log.cc:1196] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/fa92ab8198224f0cb8cf558b679aa630
I20260620 14:48:29.533697 16486 ts_tablet_manager.cc:1939] T fa92ab8198224f0cb8cf558b679aa630 P 113fb3b62558486f8265f5bc4311d8a2: Deleting consensus metadata
I20260620 14:48:29.533949 16485 ts_tablet_manager.cc:1939] T fa92ab8198224f0cb8cf558b679aa630 P d3c00e694fc84e3eaf83fd9ec6a87d0b: Deleting consensus metadata
W20260620 14:48:29.535866 16182 connection.cc:535] server connection from 127.0.0.1:35782 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 14:48:29.535913 16320 connection.cc:535] server connection from 127.0.0.1:48422 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:48:29.538136 16487 raft_consensus.cc:2258] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:29.540989 16487 ts_tablet_manager.cc:1905] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:29.543891 16487 ts_tablet_manager.cc:1918] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:29.544013 16487 log.cc:1196] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-0/wal/wals/fa92ab8198224f0cb8cf558b679aa630
I20260620 14:48:29.544400 16487 ts_tablet_manager.cc:1939] T fa92ab8198224f0cb8cf558b679aa630 P 6b5d90216c0747ecbf9f808e728294ac: Deleting consensus metadata
W20260620 14:48:29.546366 16044 connection.cc:441] server connection from 127.0.0.1:53902 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53902 (request call id 1) could send its response
I20260620 14:48:29.548918 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.527230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45450 (local address 127.8.221.253:37131)
0620 14:48:29.527451 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.527458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.538101 (+ 10643us) server_negotiation.cc:407] Connection header received
0620 14:48:29.538256 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.538261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.538328 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.538422 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:29.539557 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.540367 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.541693 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.542028 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.542104 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.544266 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:29.544273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:29.544280 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:29.544310 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:29.546903 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.547463 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.547467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.547469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.547558 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.547862 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.547866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.547868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.548071 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:29.548175 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:29.548569 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 14:48:29.548703 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260620 14:48:29.551376 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.539656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46048 (local address 127.8.221.252:37817)
0620 14:48:29.539813 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.539817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.539832 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:48:29.539884 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.539887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.539928 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.540037 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:29.541430 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.542272 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.545188 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.545545 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.545617 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.547380 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:29.547384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:29.547387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:29.547410 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:29.549144 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.549683 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.549687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.549688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.549767 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.550125 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.550128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.550130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.550618 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:29.550728 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:29.551094 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 14:48:29.551215 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:48:29.556253 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.542985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45464 (local address 127.8.221.253:37131)
0620 14:48:29.543107 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.543112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.546179 (+ 3067us) server_negotiation.cc:407] Connection header received
0620 14:48:29.546322 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.546325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.546382 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.546439 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:29.547468 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.548215 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.549377 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.549697 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.549785 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.551647 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:29.551651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:29.551655 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:29.551683 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:29.554294 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.554850 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.554854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.554856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.554948 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.555262 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.555266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.555267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.555487 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:29.555623 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:29.555952 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 14:48:29.556087 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:48:29.559355 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.527333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46034 (local address 127.8.221.252:37817)
0620 14:48:29.527561 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.527568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.550598 (+ 23030us) server_negotiation.cc:407] Connection header received
0620 14:48:29.550791 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.550795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.550856 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.550915 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:29.552018 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.552534 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.553593 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.553939 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.554051 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.555513 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:29.555517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:29.555520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:29.555544 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:29.557314 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.557729 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.557733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.557735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.557882 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:29.558180 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:29.558185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:29.558187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:29.558423 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:29.558564 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:29.558789 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 14:48:29.558922 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260620 14:48:30.508527 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:30.507968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.195:58627)
0620 14:48:30.508276 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:30.508397 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":110,"threads_started":1}
W20260620 14:48:30.509341 16431 heartbeater.cc:644] Failed to heartbeat to 127.8.221.254:41225 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
I20260620 14:48:30.509687 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:30.508603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.193:44057)
0620 14:48:30.509429 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:30.509534 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":91,"threads_started":1}
I20260620 14:48:30.509841 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:30.509323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.194:56407)
0620 14:48:30.509608 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:30.509700 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":84,"threads_started":1}
W20260620 14:48:30.510114 16295 heartbeater.cc:644] Failed to heartbeat to 127.8.221.254:41225 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
W20260620 14:48:30.510138 16158 heartbeater.cc:644] Failed to heartbeat to 127.8.221.254:41225 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
I20260620 14:48:31.024827 16498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3c4ab27b63c43d1b12d4e179d350d98)
I20260620 14:48:31.025116 16498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } }
I20260620 14:48:31.025466 16499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3c4ab27b63c43d1b12d4e179d350d98)
I20260620 14:48:31.025727 16499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } }
I20260620 14:48:31.026396 16498 leader_election.cc:288] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3c4ab27b63c43d1b12d4e179d350d98 (127.8.221.254:41225), 3e72bddb92f142bb9e99468a591005e0 (127.8.221.253:37131)
I20260620 14:48:31.027014 16499 leader_election.cc:288] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3c4ab27b63c43d1b12d4e179d350d98 (127.8.221.254:41225), 914c6ea4107a41b18d22d611649e61e2 (127.8.221.252:37817)
I20260620 14:48:31.027271 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:31.026644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:53588)
0620 14:48:31.027020 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:31.027134 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":86,"threads_started":1}
W20260620 14:48:31.027480 15964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
I20260620 14:48:31.027750 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:31.027110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:53602)
0620 14:48:31.027511 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:31.027641 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":62,"threads_started":1}
W20260620 14:48:31.028038 15779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
I20260620 14:48:31.028798 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:31.028542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:53618)
0620 14:48:31.028667 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:31.028732 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 14:48:31.028927 15964 leader_election.cc:334] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d3c4ab27b63c43d1b12d4e179d350d98 (127.8.221.254:41225): Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
I20260620 14:48:31.029165 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:31.028813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:53626)
0620 14:48:31.029029 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:31.029099 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
W20260620 14:48:31.029349 15779 leader_election.cc:334] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d3c4ab27b63c43d1b12d4e179d350d98 (127.8.221.254:41225): Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
I20260620 14:48:31.032361 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:31.026652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:37131 (local address 127.0.0.1:45474)
0620 14:48:31.027114 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:31.027131 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:31.027242 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:31.027962 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:31.027967 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:31.027986 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:31.028122 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:31.028130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.029572 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.029575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:31.031017 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:31.031030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.031945 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.031948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:31.032019 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:31.032044 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:48:31.032109 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:48:31.032221 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":83,"threads_started":1}
I20260620 14:48:31.032374 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:31.026774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.8.221.253:37131)
0620 14:48:31.027698 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:31.027704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:31.027720 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:48:31.027776 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:31.027780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:31.027828 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:31.028236 (+ 408us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:31.028347 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.029408 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.031181 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.031816 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.031939 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:31.032003 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:31.032154 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 14:48:31.032230 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":218,"server-negotiator.queue_time_us":855,"thread_start_us":61,"threads_started":1}
I20260620 14:48:31.032672 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:31.027434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37817 (local address 127.0.0.1:46052)
0620 14:48:31.027896 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:31.027911 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:31.028041 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:31.028348 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:31.028351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:31.028366 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:31.028721 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:31.028731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.029984 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.029988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:31.031273 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:31.031281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.032212 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.032216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:31.032345 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:31.032376 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:48:31.032462 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:48:31.032554 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260620 14:48:31.032753 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:31.027603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46052 (local address 127.8.221.252:37817)
0620 14:48:31.027968 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:31.027972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:31.027986 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:48:31.028166 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:31.028170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:31.028224 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:31.028835 (+ 611us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:31.029212 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.029866 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.031396 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.032066 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.032193 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:31.032248 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:31.032538 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 14:48:31.032625 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":88,"threads_started":1}
I20260620 14:48:31.032970 15803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "914c6ea4107a41b18d22d611649e61e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3e72bddb92f142bb9e99468a591005e0" is_pre_election: true
I20260620 14:48:31.033150 15984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e72bddb92f142bb9e99468a591005e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "914c6ea4107a41b18d22d611649e61e2" is_pre_election: true
I20260620 14:48:31.033245 15803 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 914c6ea4107a41b18d22d611649e61e2 in term 1.
I20260620 14:48:31.033331 15984 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e72bddb92f142bb9e99468a591005e0 in term 1.
I20260620 14:48:31.033615 15963 leader_election.cc:302] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e72bddb92f142bb9e99468a591005e0, 914c6ea4107a41b18d22d611649e61e2; no voters: d3c4ab27b63c43d1b12d4e179d350d98
I20260620 14:48:31.033707 15779 leader_election.cc:302] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e72bddb92f142bb9e99468a591005e0, 914c6ea4107a41b18d22d611649e61e2; no voters: d3c4ab27b63c43d1b12d4e179d350d98
I20260620 14:48:31.033898 16498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 14:48:31.033987 16498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3c4ab27b63c43d1b12d4e179d350d98)
I20260620 14:48:31.034004 16499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 14:48:31.034032 16498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:48:31.034062 16499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3c4ab27b63c43d1b12d4e179d350d98)
I20260620 14:48:31.034111 16499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:48:31.035334 16498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } }
I20260620 14:48:31.035368 16499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } }
I20260620 14:48:31.035725 16499 leader_election.cc:288] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [CANDIDATE]: Term 2 election: Requested vote from peers d3c4ab27b63c43d1b12d4e179d350d98 (127.8.221.254:41225), 914c6ea4107a41b18d22d611649e61e2 (127.8.221.252:37817)
I20260620 14:48:31.035739 16498 leader_election.cc:288] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [CANDIDATE]: Term 2 election: Requested vote from peers d3c4ab27b63c43d1b12d4e179d350d98 (127.8.221.254:41225), 3e72bddb92f142bb9e99468a591005e0 (127.8.221.253:37131)
I20260620 14:48:31.036144 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:31.035847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:53632)
0620 14:48:31.035969 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:31.036063 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 14:48:31.036175 15984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e72bddb92f142bb9e99468a591005e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "914c6ea4107a41b18d22d611649e61e2"
I20260620 14:48:31.036376 15984 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 3e72bddb92f142bb9e99468a591005e0 in current term 2: Already voted for candidate 914c6ea4107a41b18d22d611649e61e2 in this term.
I20260620 14:48:31.036554 15803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "914c6ea4107a41b18d22d611649e61e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3e72bddb92f142bb9e99468a591005e0"
I20260620 14:48:31.036712 15803 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 914c6ea4107a41b18d22d611649e61e2 in current term 2: Already voted for candidate 3e72bddb92f142bb9e99468a591005e0 in this term.
I20260620 14:48:31.037262 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:31.036710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:53656)
0620 14:48:31.037140 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:31.037188 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368}
I20260620 14:48:31.037950 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:31.037707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:53668)
0620 14:48:31.037839 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:31.037886 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 14:48:31.037950 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:31.037658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:53664)
0620 14:48:31.037827 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:31.037887 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 14:48:31.038112 15964 leader_election.cc:334] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d3c4ab27b63c43d1b12d4e179d350d98 (127.8.221.254:41225): Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
I20260620 14:48:31.038198 15964 leader_election.cc:302] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 914c6ea4107a41b18d22d611649e61e2; no voters: 3e72bddb92f142bb9e99468a591005e0, d3c4ab27b63c43d1b12d4e179d350d98
I20260620 14:48:31.038324 16498 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260620 14:48:31.038479 15779 leader_election.cc:334] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d3c4ab27b63c43d1b12d4e179d350d98 (127.8.221.254:41225): Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
I20260620 14:48:31.038552 15779 leader_election.cc:302] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e72bddb92f142bb9e99468a591005e0; no voters: 914c6ea4107a41b18d22d611649e61e2, d3c4ab27b63c43d1b12d4e179d350d98
I20260620 14:48:31.038663 16499 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 14:48:31.510675 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:31.509991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.195:44067)
0620 14:48:31.510422 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:31.510529 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":105,"threads_started":1}
I20260620 14:48:31.511047 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:31.510636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.194:49613)
0620 14:48:31.510862 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:31.510932 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
I20260620 14:48:31.511328 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:31.510822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.193:56915)
0620 14:48:31.511158 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:31.511220 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":110,"threads_started":1}
I20260620 14:48:32.512125 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:32.511531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.195:60725)
0620 14:48:32.511888 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:32.511965 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":96,"threads_started":1}
I20260620 14:48:32.512192 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:32.511734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.194:60543)
0620 14:48:32.512008 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:32.512074 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":118,"threads_started":1}
I20260620 14:48:32.512400 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:32.511914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.193:55393)
0620 14:48:32.512143 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:32.512202 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":76,"threads_started":1}
I20260620 14:48:33.058506 16514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:33.058753 16514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } }
I20260620 14:48:33.059309 16514 leader_election.cc:288] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d3c4ab27b63c43d1b12d4e179d350d98 (127.8.221.254:41225), 914c6ea4107a41b18d22d611649e61e2 (127.8.221.252:37817)
I20260620 14:48:33.060137 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.059576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:53986)
0620 14:48:33.059929 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.059999 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
I20260620 14:48:33.060333 15984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e72bddb92f142bb9e99468a591005e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "914c6ea4107a41b18d22d611649e61e2" is_pre_election: true
I20260620 14:48:33.060539 15984 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e72bddb92f142bb9e99468a591005e0 in term 2.
I20260620 14:48:33.061007 15779 leader_election.cc:302] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e72bddb92f142bb9e99468a591005e0, 914c6ea4107a41b18d22d611649e61e2; no voters:
I20260620 14:48:33.061133 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.060850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:54002)
0620 14:48:33.061009 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.061068 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 14:48:33.061177 16514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 14:48:33.061228 16514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:33.061275 16514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 2 FOLLOWER]: Advancing to term 3
W20260620 14:48:33.061363 15779 leader_election.cc:334] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d3c4ab27b63c43d1b12d4e179d350d98 (127.8.221.254:41225): Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
I20260620 14:48:33.062657 16514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } }
I20260620 14:48:33.062983 16514 leader_election.cc:288] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [CANDIDATE]: Term 3 election: Requested vote from peers d3c4ab27b63c43d1b12d4e179d350d98 (127.8.221.254:41225), 914c6ea4107a41b18d22d611649e61e2 (127.8.221.252:37817)
I20260620 14:48:33.063366 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.063115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:54008)
0620 14:48:33.063224 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.063306 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 14:48:33.063581 15984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e72bddb92f142bb9e99468a591005e0" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "914c6ea4107a41b18d22d611649e61e2"
I20260620 14:48:33.063753 15984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:48:33.064159 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.063869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:54018)
0620 14:48:33.064034 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.064101 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260620 14:48:33.064294 15779 leader_election.cc:334] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d3c4ab27b63c43d1b12d4e179d350d98 (127.8.221.254:41225): Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
I20260620 14:48:33.064978 15984 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e72bddb92f142bb9e99468a591005e0 in term 3.
I20260620 14:48:33.065349 15779 leader_election.cc:302] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e72bddb92f142bb9e99468a591005e0, 914c6ea4107a41b18d22d611649e61e2; no voters: d3c4ab27b63c43d1b12d4e179d350d98
I20260620 14:48:33.065534 16514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:48:33.065717 16514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [term 3 LEADER]: Becoming Leader. State: Replica: 3e72bddb92f142bb9e99468a591005e0, State: Running, Role: LEADER
I20260620 14:48:33.066025 16514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } }
I20260620 14:48:33.067620 16517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e72bddb92f142bb9e99468a591005e0. Latest consensus state: current_term: 3 leader_uuid: "3e72bddb92f142bb9e99468a591005e0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } } }
I20260620 14:48:33.067781 16517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:33.068138 16519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:33.070454 16519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:33.070843 16519 catalog_manager.cc:1259] Loaded cluster ID: 1337fddfcbd84c00afdea2e078edfce1
I20260620 14:48:33.070891 16519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:33.071444 16519 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:33.071836 16519 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: Loaded TSK: 0
I20260620 14:48:33.072528 16519 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:48:33.137866 15790 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45464:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:48:33.141939 15790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:33.145720 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.145430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:54028)
0620 14:48:33.145579 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.145632 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 14:48:33.145889 15984 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [term 3 FOLLOWER]: Refusing update from remote peer 3e72bddb92f142bb9e99468a591005e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 14:48:33.146482 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.146258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.0.0.1:54040)
0620 14:48:33.146365 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.146418 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 14:48:33.146610 16514 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 14:48:33.146855 15779 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 -> Peer d3c4ab27b63c43d1b12d4e179d350d98 (127.8.221.254:41225): Couldn't send request to peer d3c4ab27b63c43d1b12d4e179d350d98. Status: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 14:48:33.148452 16521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e72bddb92f142bb9e99468a591005e0. Latest consensus state: current_term: 3 leader_uuid: "3e72bddb92f142bb9e99468a591005e0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } } }
I20260620 14:48:33.148628 16521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:48:33.149919 16514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "3e72bddb92f142bb9e99468a591005e0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } } }
I20260620 14:48:33.150074 16514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:33.150164 16521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "3e72bddb92f142bb9e99468a591005e0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c4ab27b63c43d1b12d4e179d350d98" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 41225 } } peers { permanent_uuid: "3e72bddb92f142bb9e99468a591005e0" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 37131 } attrs { promote: false } } peers { permanent_uuid: "914c6ea4107a41b18d22d611649e61e2" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37817 } attrs { promote: false } } }
I20260620 14:48:33.150306 16521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 914c6ea4107a41b18d22d611649e61e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:48:33.150426 16522 mvcc.cc:205] Tried to move back new op lower bound from 7298936476239257600 to 7298936475921305600. Current Snapshot: MvccSnapshot[applied={T|T < 7298936476239257600}]
I20260620 14:48:33.162716 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.157764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:38715 (local address 127.0.0.1:35810)
0620 14:48:33.158086 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.158100 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:33.158196 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:33.158667 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:33.158669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:33.158683 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:33.158801 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:33.158808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.159803 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.159809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:33.161051 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:33.161061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.162220 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.162224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:33.162317 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:33.162356 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:48:33.162442 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:48:33.162544 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":4,"thread_start_us":146,"threads_started":1}
I20260620 14:48:33.162973 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.158102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35810 (local address 127.8.221.194:38715)
0620 14:48:33.158392 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:33.158398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:33.158414 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:48:33.158493 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:33.158497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:33.158559 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:33.158652 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:33.158964 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.159690 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.161216 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.162023 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.162219 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:33.162299 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:33.162697 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 14:48:33.162795 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":103,"threads_started":1}
I20260620 14:48:33.163383 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.158175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:34701 (local address 127.0.0.1:48430)
0620 14:48:33.158397 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.158410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:33.158492 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:33.159000 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:33.159003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:33.159019 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:33.159089 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:33.159094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.160334 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.160337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:33.161547 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:33.161555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.163027 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.163031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:33.163111 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:33.163131 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:48:33.163196 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:48:33.163265 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":84,"thread_start_us":120,"threads_started":1}
I20260620 14:48:33.163520 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.158274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48430 (local address 127.8.221.195:34701)
0620 14:48:33.158527 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:33.158534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:33.158552 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:48:33.158613 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:33.158617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:33.158668 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:33.158773 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:33.159227 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.160172 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.161677 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.162319 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.162436 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:33.162553 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:33.163270 (+ 717us) server_negotiation.cc:299] Negotiation successful
0620 14:48:33.163357 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":60,"threads_started":1}
I20260620 14:48:33.163612 16228 tablet_service.cc:1467] Processing CreateTablet for tablet f57a53035fff41b9be855a33afd9f221 (DEFAULT_TABLE table=test-table [id=7cd8816fa68841a1823b940940922e08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:33.163805 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.157347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:41493 (local address 127.0.0.1:53916)
0620 14:48:33.157466 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.157481 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:33.157575 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:33.159045 (+ 1470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:33.159048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:33.159067 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:33.159183 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:33.159190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.160343 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.160347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:33.161722 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:33.161733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.163411 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.163415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:33.163506 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:33.163534 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:48:33.163589 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:48:33.163675 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":94}
I20260620 14:48:33.163962 16228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57a53035fff41b9be855a33afd9f221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:33.164220 16364 tablet_service.cc:1467] Processing CreateTablet for tablet f57a53035fff41b9be855a33afd9f221 (DEFAULT_TABLE table=test-table [id=7cd8816fa68841a1823b940940922e08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:33.164434 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.157744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53916 (local address 127.8.221.193:41493)
0620 14:48:33.158734 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:33.158741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:33.158757 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:48:33.158811 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:33.158814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:33.158862 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:33.159030 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:33.159321 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.160196 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.162584 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.163245 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.164072 (+ 827us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:33.164141 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:33.164177 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 14:48:33.164249 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":893,"thread_start_us":182,"threads_started":1}
I20260620 14:48:33.164492 16364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57a53035fff41b9be855a33afd9f221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:33.165066 16091 tablet_service.cc:1467] Processing CreateTablet for tablet f57a53035fff41b9be855a33afd9f221 (DEFAULT_TABLE table=test-table [id=7cd8816fa68841a1823b940940922e08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:33.165354 16091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f57a53035fff41b9be855a33afd9f221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:33.166922 16532 tablet_bootstrap.cc:492] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b: Bootstrap starting.
I20260620 14:48:33.168088 16533 tablet_bootstrap.cc:492] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2: Bootstrap starting.
I20260620 14:48:33.168288 16532 tablet_bootstrap.cc:654] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:33.168339 16534 tablet_bootstrap.cc:492] T f57a53035fff41b9be855a33afd9f221 P 6b5d90216c0747ecbf9f808e728294ac: Bootstrap starting.
I20260620 14:48:33.169353 16533 tablet_bootstrap.cc:654] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:33.169404 16534 tablet_bootstrap.cc:654] T f57a53035fff41b9be855a33afd9f221 P 6b5d90216c0747ecbf9f808e728294ac: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:33.170014 16532 tablet_bootstrap.cc:492] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b: No bootstrap required, opened a new log
I20260620 14:48:33.170125 16532 ts_tablet_manager.cc:1397] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:33.170702 16533 tablet_bootstrap.cc:492] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2: No bootstrap required, opened a new log
I20260620 14:48:33.170651 16532 raft_consensus.cc:348] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } } peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } }
I20260620 14:48:33.170780 16532 raft_consensus.cc:374] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:33.170794 16533 ts_tablet_manager.cc:1397] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:33.170830 16532 raft_consensus.cc:729] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c00e694fc84e3eaf83fd9ec6a87d0b, State: Initialized, Role: FOLLOWER
I20260620 14:48:33.170871 16534 tablet_bootstrap.cc:492] T f57a53035fff41b9be855a33afd9f221 P 6b5d90216c0747ecbf9f808e728294ac: No bootstrap required, opened a new log
I20260620 14:48:33.170984 16534 ts_tablet_manager.cc:1397] T f57a53035fff41b9be855a33afd9f221 P 6b5d90216c0747ecbf9f808e728294ac: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 14:48:33.170985 16532 consensus_queue.cc:260] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } } peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } }
I20260620 14:48:33.171265 16533 raft_consensus.cc:348] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } } peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } }
I20260620 14:48:33.171372 16533 raft_consensus.cc:374] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:33.171379 16532 ts_tablet_manager.cc:1428] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:33.171411 16533 raft_consensus.cc:729] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 113fb3b62558486f8265f5bc4311d8a2, State: Initialized, Role: FOLLOWER
I20260620 14:48:33.171540 16533 consensus_queue.cc:260] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } } peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } }
I20260620 14:48:33.171571 16534 raft_consensus.cc:348] T f57a53035fff41b9be855a33afd9f221 P 6b5d90216c0747ecbf9f808e728294ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } } peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } }
I20260620 14:48:33.171701 16534 raft_consensus.cc:374] T f57a53035fff41b9be855a33afd9f221 P 6b5d90216c0747ecbf9f808e728294ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:33.171746 16534 raft_consensus.cc:729] T f57a53035fff41b9be855a33afd9f221 P 6b5d90216c0747ecbf9f808e728294ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b5d90216c0747ecbf9f808e728294ac, State: Initialized, Role: FOLLOWER
I20260620 14:48:33.171840 16533 ts_tablet_manager.cc:1428] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:33.171917 16534 consensus_queue.cc:260] T f57a53035fff41b9be855a33afd9f221 P 6b5d90216c0747ecbf9f808e728294ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } } peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } }
I20260620 14:48:33.172271 16534 ts_tablet_manager.cc:1428] T f57a53035fff41b9be855a33afd9f221 P 6b5d90216c0747ecbf9f808e728294ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260620 14:48:33.173596 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.173089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.193:57471)
0620 14:48:33.173386 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.173468 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":109,"threads_started":1}
I20260620 14:48:33.175962 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.172521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.195:57753)
0620 14:48:33.175751 (+ 3230us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.175834 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3134,"thread_start_us":108,"threads_started":1}
I20260620 14:48:33.176494 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.173039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.194:52865)
0620 14:48:33.176302 (+ 3263us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.176369 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3189,"thread_start_us":127,"threads_started":1}
I20260620 14:48:33.176779 16432 heartbeater.cc:503] Master 127.8.221.253:37131 was elected leader, sending a full tablet report...
I20260620 14:48:33.177634 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.177339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.193:40079)
0620 14:48:33.177474 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.177541 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 14:48:33.178858 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.178611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.194:38965)
0620 14:48:33.178734 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.178783 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 14:48:33.178877 16159 heartbeater.cc:503] Master 127.8.221.253:37131 was elected leader, sending a full tablet report...
I20260620 14:48:33.179301 16296 heartbeater.cc:503] Master 127.8.221.253:37131 was elected leader, sending a full tablet report...
I20260620 14:48:33.179385 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.176924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.195:45047)
0620 14:48:33.179227 (+ 2303us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.179297 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 14:48:33.258591 16538 raft_consensus.cc:482] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:33.258777 16538 raft_consensus.cc:504] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } } peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } }
I20260620 14:48:33.259425 16538 leader_election.cc:288] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b5d90216c0747ecbf9f808e728294ac (127.8.221.193:41493), 113fb3b62558486f8265f5bc4311d8a2 (127.8.221.195:34701)
I20260620 14:48:33.259895 16111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57a53035fff41b9be855a33afd9f221" candidate_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b5d90216c0747ecbf9f808e728294ac" is_pre_election: true
I20260620 14:48:33.260030 16384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57a53035fff41b9be855a33afd9f221" candidate_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113fb3b62558486f8265f5bc4311d8a2" is_pre_election: true
I20260620 14:48:33.260133 16111 raft_consensus.cc:2454] T f57a53035fff41b9be855a33afd9f221 P 6b5d90216c0747ecbf9f808e728294ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3c00e694fc84e3eaf83fd9ec6a87d0b in term 0.
I20260620 14:48:33.260196 16384 raft_consensus.cc:2454] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3c00e694fc84e3eaf83fd9ec6a87d0b in term 0.
I20260620 14:48:33.260548 16184 leader_election.cc:302] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 113fb3b62558486f8265f5bc4311d8a2, 6b5d90216c0747ecbf9f808e728294ac, d3c00e694fc84e3eaf83fd9ec6a87d0b; no voters:
I20260620 14:48:33.260752 16538 raft_consensus.cc:2781] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:33.260813 16538 raft_consensus.cc:482] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:33.260878 16538 raft_consensus.cc:3037] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:33.262974 16538 raft_consensus.cc:504] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } } peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } }
I20260620 14:48:33.263361 16538 leader_election.cc:288] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [CANDIDATE]: Term 1 election: Requested vote from peers 6b5d90216c0747ecbf9f808e728294ac (127.8.221.193:41493), 113fb3b62558486f8265f5bc4311d8a2 (127.8.221.195:34701)
I20260620 14:48:33.263820 16384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57a53035fff41b9be855a33afd9f221" candidate_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113fb3b62558486f8265f5bc4311d8a2"
I20260620 14:48:33.263867 16111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f57a53035fff41b9be855a33afd9f221" candidate_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b5d90216c0747ecbf9f808e728294ac"
I20260620 14:48:33.263978 16384 raft_consensus.cc:3037] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2 [term 0 FOLLOWER]: Advancing to term 1
W20260620 14:48:33.263964 16160 tablet.cc:2369] T f57a53035fff41b9be855a33afd9f221 P 6b5d90216c0747ecbf9f808e728294ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:48:33.264047 16111 raft_consensus.cc:3037] T f57a53035fff41b9be855a33afd9f221 P 6b5d90216c0747ecbf9f808e728294ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:33.264968 16111 raft_consensus.cc:2454] T f57a53035fff41b9be855a33afd9f221 P 6b5d90216c0747ecbf9f808e728294ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3c00e694fc84e3eaf83fd9ec6a87d0b in term 1.
I20260620 14:48:33.265166 16384 raft_consensus.cc:2454] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3c00e694fc84e3eaf83fd9ec6a87d0b in term 1.
I20260620 14:48:33.265391 16182 leader_election.cc:302] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b5d90216c0747ecbf9f808e728294ac, d3c00e694fc84e3eaf83fd9ec6a87d0b; no voters:
I20260620 14:48:33.265547 16538 raft_consensus.cc:2781] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:33.265774 16538 raft_consensus.cc:686] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 1 LEADER]: Becoming Leader. State: Replica: d3c00e694fc84e3eaf83fd9ec6a87d0b, State: Running, Role: LEADER
I20260620 14:48:33.266004 16538 consensus_queue.cc:237] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } } peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } }
I20260620 14:48:33.266636 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.266338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.194:41795)
0620 14:48:33.266475 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.266543 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 14:48:33.268100 15790 catalog_manager.cc:5495] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b reported cstate change: term changed from 0 to 1, leader changed from <none> to d3c00e694fc84e3eaf83fd9ec6a87d0b (127.8.221.194). New cstate: current_term: 1 leader_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3c00e694fc84e3eaf83fd9ec6a87d0b" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 } health_report { overall_health: UNKNOWN } } }
I20260620 14:48:33.291926 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.288466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35814 (local address 127.8.221.194:38715)
0620 14:48:33.288614 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:33.288619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:33.288634 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:48:33.288689 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:33.288692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:33.288742 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:33.288837 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:33.289053 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.289783 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.290725 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.290987 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.291082 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:33.291361 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:33.291446 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:33.291510 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:33.291633 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 14:48:33.291722 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:48:33.294943 16384 raft_consensus.cc:1261] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2 [term 1 FOLLOWER]: Refusing update from remote peer d3c00e694fc84e3eaf83fd9ec6a87d0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:33.294996 16111 raft_consensus.cc:1261] T f57a53035fff41b9be855a33afd9f221 P 6b5d90216c0747ecbf9f808e728294ac [term 1 FOLLOWER]: Refusing update from remote peer d3c00e694fc84e3eaf83fd9ec6a87d0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:33.295466 16538 consensus_queue.cc:1035] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "113fb3b62558486f8265f5bc4311d8a2" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:33.295629 16544 consensus_queue.cc:1035] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b5d90216c0747ecbf9f808e728294ac" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:33.298404 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.297666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.195:58185)
0620 14:48:33.298216 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.298294 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462}
I20260620 14:48:33.299134 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.298840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.193:56275)
0620 14:48:33.298999 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.299055 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 14:48:33.302031 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.301458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.194:47393)
0620 14:48:33.301879 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.301941 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324}
I20260620 14:48:33.303613 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.303274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.193:34989)
0620 14:48:33.303469 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.303524 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260620 14:48:33.304328 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.303998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:41225 (local address 127.8.221.195:51353)
0620 14:48:33.304182 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.304244 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 14:48:33.308353 15793 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:33.308488 15793 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45464:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:33.312590 15793 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e72bddb92f142bb9e99468a591005e0: Sending DeleteTablet for 3 replicas of tablet f57a53035fff41b9be855a33afd9f221
I20260620 14:48:33.313300 16228 tablet_service.cc:1514] Processing DeleteTablet for tablet f57a53035fff41b9be855a33afd9f221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:33 UTC) from {username='oryx', principal='oryx/127.8.221.253@KRBTEST.COM'} at 127.0.0.1:35810
I20260620 14:48:33.313385 16091 tablet_service.cc:1514] Processing DeleteTablet for tablet f57a53035fff41b9be855a33afd9f221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:33 UTC) from {username='oryx', principal='oryx/127.8.221.253@KRBTEST.COM'} at 127.0.0.1:53916
I20260620 14:48:33.313378 16364 tablet_service.cc:1514] Processing DeleteTablet for tablet f57a53035fff41b9be855a33afd9f221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:33 UTC) from {username='oryx', principal='oryx/127.8.221.253@KRBTEST.COM'} at 127.0.0.1:48430
I20260620 14:48:33.313742 16556 tablet_replica.cc:330] T f57a53035fff41b9be855a33afd9f221 P 6b5d90216c0747ecbf9f808e728294ac: stopping tablet replica
I20260620 14:48:33.313871 16557 tablet_replica.cc:330] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2: stopping tablet replica
I20260620 14:48:33.313920 16556 raft_consensus.cc:2229] T f57a53035fff41b9be855a33afd9f221 P 6b5d90216c0747ecbf9f808e728294ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:33.314038 16557 raft_consensus.cc:2229] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:33.314081 16556 raft_consensus.cc:2258] T f57a53035fff41b9be855a33afd9f221 P 6b5d90216c0747ecbf9f808e728294ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:33.314085 16555 tablet_replica.cc:330] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b: stopping tablet replica
I20260620 14:48:33.314174 16557 raft_consensus.cc:2258] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:33.314209 16555 raft_consensus.cc:2229] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:33.314437 16555 raft_consensus.cc:2258] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:33.314584 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 15937
I20260620 14:48:33.315258 16557 ts_tablet_manager.cc:1905] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:33.315876 16555 ts_tablet_manager.cc:1905] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:33.318315 16557 ts_tablet_manager.cc:1918] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:33.318418 16557 log.cc:1196] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/f57a53035fff41b9be855a33afd9f221
I20260620 14:48:33.318686 16557 ts_tablet_manager.cc:1939] T f57a53035fff41b9be855a33afd9f221 P 113fb3b62558486f8265f5bc4311d8a2: Deleting consensus metadata
I20260620 14:48:33.319329 15780 catalog_manager.cc:4841] TS 113fb3b62558486f8265f5bc4311d8a2 (127.8.221.195:34701): tablet f57a53035fff41b9be855a33afd9f221 (table test-table [id=7cd8816fa68841a1823b940940922e08]) successfully deleted
I20260620 14:48:33.319442 16555 ts_tablet_manager.cc:1918] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:33.319533 16555 log.cc:1196] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/f57a53035fff41b9be855a33afd9f221
I20260620 14:48:33.319871 16555 ts_tablet_manager.cc:1939] T f57a53035fff41b9be855a33afd9f221 P d3c00e694fc84e3eaf83fd9ec6a87d0b: Deleting consensus metadata
I20260620 14:48:33.320653 15779 catalog_manager.cc:4841] TS d3c00e694fc84e3eaf83fd9ec6a87d0b (127.8.221.194:38715): tablet f57a53035fff41b9be855a33afd9f221 (table test-table [id=7cd8816fa68841a1823b940940922e08]) successfully deleted
I20260620 14:48:33.324002 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 16165
I20260620 14:48:33.324375 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.324055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:41493 (local address 127.0.0.1:53932)
0620 14:48:33.324203 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.324266 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.193:41493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 14:48:33.324759 15778 catalog_manager.cc:4584] TS 6b5d90216c0747ecbf9f808e728294ac (127.8.221.193:41493): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f57a53035fff41b9be855a33afd9f221: Network error: Client connection negotiation failed: client connection to 127.8.221.193:41493: connect: Connection refused (error 111)
I20260620 14:48:33.332468 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 16301
I20260620 14:48:33.339908 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 15656
I20260620 14:48:33.346830 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 15757
2026-06-20T14:48:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7836 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:33 dist-test-slave-65tz krb5kdc[16560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:33 dist-test-slave-65tz krb5kdc[16560](info): set up 2 sockets
Jun 20 14:48:33 dist-test-slave-65tz krb5kdc[16560](info): commencing operation
krb5kdc: starting...
W20260620 14:48:34.442834 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:48:34 dist-test-slave-65tz krb5kdc[16560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.254.keytab.
Entry for principal oryx/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.254.keytab.
I20260620 14:48:34.575006 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:33969
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.254.keytab
--principal=oryx/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:33969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:34.689827 16573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:34.690152 16573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:34.690227 16573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:34.694219 16573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:34.694308 16573 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:48:34.694340 16573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:34.694380 16573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:34.694417 16573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:34.694450 16573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:34.699663 16573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:33969
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.254.keytab
--principal=oryx/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:33969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:34.700716 16573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:34.701728 16573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:34.711248 16581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:34.711261 16578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:34.711249 16579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:34.711289 16573 server_base.cc:1034] running on GCE node
I20260620 14:48:34.712175 16573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:34.713326 16573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:34.714568 16573 hybrid_clock.cc:648] HybridClock initialized: now 1781966914714544 us; error 45 us; skew 500 ppm
Jun 20 14:48:34 dist-test-slave-65tz krb5kdc[16560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966914, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:34.717559 16573 init.cc:377] Logged in from keytab as oryx/127.8.221.254@KRBTEST.COM (short username oryx)
I20260620 14:48:34.718595 16573 webserver.cc:460] Webserver started at http://127.8.221.254:44753/ using document root <none> and password file <none>
I20260620 14:48:34.719193 16573 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:34.719260 16573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:34.719497 16573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:34.721679 16573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "d601664b05e64c2085b8323d3f39a885"
format_stamp: "Formatted at 2026-06-20 14:48:34 on dist-test-slave-65tz"
server_key: "1bd050333ec5f95e17b0f811e0b4a904"
server_key_iv: "4f97a7388f829918285214926750b7cf"
server_key_version: "encryptionkey@0"
I20260620 14:48:34.722340 16573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "d601664b05e64c2085b8323d3f39a885"
format_stamp: "Formatted at 2026-06-20 14:48:34 on dist-test-slave-65tz"
server_key: "1bd050333ec5f95e17b0f811e0b4a904"
server_key_iv: "4f97a7388f829918285214926750b7cf"
server_key_version: "encryptionkey@0"
I20260620 14:48:34.726541 16573 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:34.729454 16588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:34.730959 16573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:34.731123 16573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "d601664b05e64c2085b8323d3f39a885"
format_stamp: "Formatted at 2026-06-20 14:48:34 on dist-test-slave-65tz"
server_key: "1bd050333ec5f95e17b0f811e0b4a904"
server_key_iv: "4f97a7388f829918285214926750b7cf"
server_key_version: "encryptionkey@0"
I20260620 14:48:34.731253 16573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:34.753993 16573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:34.757283 16573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:34.757503 16573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:34.766512 16573 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:33969
I20260620 14:48:34.766623 16650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:33969 every 8 connection(s)
I20260620 14:48:34.767531 16573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:34.771281 16651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:34.772636 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 16573
I20260620 14:48:34.772815 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:48:34.773162 9079 external_mini_cluster.cc:1428] Setting key 31fa7a1914efd3743d9ad23bca9e832e
I20260620 14:48:34.779666 16651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885: Bootstrap starting.
I20260620 14:48:34.783625 16651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:34.784770 16651 log.cc:824] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:48:34 dist-test-slave-65tz krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM
I20260620 14:48:34.787606 16651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885: No bootstrap required, opened a new log
I20260620 14:48:34.792021 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:34.774889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54392 (local address 127.8.221.254:33969)
0620 14:48:34.775355 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:34.775372 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:34.775407 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:48:34.776220 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:34.776239 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:34.776600 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:34.776896 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:34.778054 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:34.779804 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:34.781594 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:34.782325 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:34.782503 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:34.786127 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:34.786133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:34.786152 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:34.786181 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:34.788866 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:34.789324 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:34.789330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:34.789336 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:34.789468 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:34.789743 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:34.789747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:34.789748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:34.790258 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:34.790451 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:34.790742 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 14:48:34.791041 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":165,"threads_started":1}
I20260620 14:48:34.791965 16651 raft_consensus.cc:348] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d601664b05e64c2085b8323d3f39a885" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33969 } }
I20260620 14:48:34.792243 16651 raft_consensus.cc:374] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:34.792320 16651 raft_consensus.cc:729] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d601664b05e64c2085b8323d3f39a885, State: Initialized, Role: FOLLOWER
I20260620 14:48:34.792778 16651 consensus_queue.cc:260] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d601664b05e64c2085b8323d3f39a885" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33969 } }
I20260620 14:48:34.792928 16651 raft_consensus.cc:388] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:34.792999 16651 raft_consensus.cc:482] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:34.793107 16651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:34.794169 16651 raft_consensus.cc:504] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d601664b05e64c2085b8323d3f39a885" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33969 } }
I20260620 14:48:34.794539 16651 leader_election.cc:302] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d601664b05e64c2085b8323d3f39a885; no voters:
I20260620 14:48:34.794910 16651 leader_election.cc:288] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:34.794974 16656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:34.795166 16656 raft_consensus.cc:686] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885 [term 1 LEADER]: Becoming Leader. State: Replica: d601664b05e64c2085b8323d3f39a885, State: Running, Role: LEADER
I20260620 14:48:34.795452 16656 consensus_queue.cc:237] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d601664b05e64c2085b8323d3f39a885" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33969 } }
I20260620 14:48:34.796168 16651 sys_catalog.cc:564] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:34.797258 16657 sys_catalog.cc:455] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d601664b05e64c2085b8323d3f39a885. Latest consensus state: current_term: 1 leader_uuid: "d601664b05e64c2085b8323d3f39a885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d601664b05e64c2085b8323d3f39a885" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33969 } } }
I20260620 14:48:34.797446 16657 sys_catalog.cc:458] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:34.797816 16658 sys_catalog.cc:455] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d601664b05e64c2085b8323d3f39a885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d601664b05e64c2085b8323d3f39a885" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33969 } } }
I20260620 14:48:34.797914 16658 sys_catalog.cc:458] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:34.798046 16665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:34.801255 16665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:34.811532 16665 catalog_manager.cc:1347] Generated new cluster ID: a8b6c478a30c494e85606deeff2d2045
I20260620 14:48:34.811612 16665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:34.832908 16665 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:34.834235 16665 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:34.842680 16665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885: Generated new TSK 0
I20260620 14:48:34.843561 16665 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.193.keytab.
Entry for principal oryx/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.193.keytab.
I20260620 14:48:34.884977 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33969
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.193.keytab
--principal=oryx/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:35.016959 16677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:35.017258 16677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:35.017309 16677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:35.021761 16677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:35.021852 16677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:35.021896 16677 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:48:35.021934 16677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:48:35.027602 16677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.193.keytab
--principal=oryx/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.221.254:33969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:35.028626 16677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:35.029628 16677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:35.039574 16682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:35.039574 16683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:35.039566 16685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:35.039898 16677 server_base.cc:1034] running on GCE node
I20260620 14:48:35.040426 16677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:35.041266 16677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:35.042460 16677 hybrid_clock.cc:648] HybridClock initialized: now 1781966915042434 us; error 42 us; skew 500 ppm
Jun 20 14:48:35 dist-test-slave-65tz krb5kdc[16560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966915, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:35.045619 16677 init.cc:377] Logged in from keytab as oryx/127.8.221.193@KRBTEST.COM (short username oryx)
I20260620 14:48:35.046684 16677 webserver.cc:460] Webserver started at http://127.8.221.193:37371/ using document root <none> and password file <none>
I20260620 14:48:35.047366 16677 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:35.047469 16677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:35.047735 16677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:35.050148 16677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "fa4709532c534b7ab9d6911dda04a269"
format_stamp: "Formatted at 2026-06-20 14:48:35 on dist-test-slave-65tz"
server_key: "eb8587fc7df16f5343bfc32da65bad7d"
server_key_iv: "4babc2c160630f009190262caa826e97"
server_key_version: "encryptionkey@0"
I20260620 14:48:35.050788 16677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "fa4709532c534b7ab9d6911dda04a269"
format_stamp: "Formatted at 2026-06-20 14:48:35 on dist-test-slave-65tz"
server_key: "eb8587fc7df16f5343bfc32da65bad7d"
server_key_iv: "4babc2c160630f009190262caa826e97"
server_key_version: "encryptionkey@0"
I20260620 14:48:35.055298 16677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 14:48:35.058148 16692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:35.059537 16677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:48:35.059679 16677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "fa4709532c534b7ab9d6911dda04a269"
format_stamp: "Formatted at 2026-06-20 14:48:35 on dist-test-slave-65tz"
server_key: "eb8587fc7df16f5343bfc32da65bad7d"
server_key_iv: "4babc2c160630f009190262caa826e97"
server_key_version: "encryptionkey@0"
I20260620 14:48:35.059795 16677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:35.072618 16677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:35.076180 16677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:35.076431 16677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:35.078199 16677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:35.078279 16677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:35.078356 16677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:35.078399 16677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:48:35 dist-test-slave-65tz krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966915, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM
I20260620 14:48:35.091923 16677 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:40715
I20260620 14:48:35.092054 16807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:40715 every 8 connection(s)
I20260620 14:48:35.093264 16677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:35.094022 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 16677
I20260620 14:48:35.094188 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:48:35.094651 9079 external_mini_cluster.cc:1428] Setting key c1afadd657db45796995e9078c718757
I20260620 14:48:35.096096 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.079357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:39913 (local address 127.8.221.254:33969)
0620 14:48:35.079503 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.079507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.080845 (+ 1338us) server_negotiation.cc:407] Connection header received
0620 14:48:35.082134 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.082137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.082193 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.082301 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:35.083648 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.084363 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.085717 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.086047 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.086164 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.090623 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:35.090628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:35.090630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:35.090662 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:35.092956 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:35.093889 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:35.093894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:35.093896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:35.094036 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:35.094552 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:35.094556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:35.094558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:35.094871 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:35.094968 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:35.095670 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 14:48:35.095873 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:48:35.097239 16701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.079982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:33969 (local address 127.8.221.193:39913)
0620 14:48:35.080644 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:35.080707 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:35.081885 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:35.082490 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:35.082498 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:35.082883 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:35.083423 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.083437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.084496 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.084500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.085587 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.085597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.086210 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.086214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.086454 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.087307 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:35.087334 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:35.090273 (+ 2939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:35.093158 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:35.093165 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:35.093188 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:35.093631 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:35.094196 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:35.094203 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:35.094207 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:35.094418 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:35.095021 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:35.095030 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:35.095364 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 14:48:35.095610 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 14:48:35.096067 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":184,"threads_started":1}
I20260620 14:48:35.098937 16808 heartbeater.cc:347] Connected to a master server at 127.8.221.254:33969
I20260620 14:48:35.099126 16808 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:35.099798 16808 heartbeater.cc:511] Master 127.8.221.254:33969 requested a full tablet report, sending...
I20260620 14:48:35.101962 16605 ts_manager.cc:194] Registered new tserver with Master: fa4709532c534b7ab9d6911dda04a269 (127.8.221.193:40715)
I20260620 14:48:35.103906 16605 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:39913
WARNING: no policy specified for oryx/127.8.221.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.221.194@KRBTEST.COM" created.
I20260620 14:48:35.110765 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.101700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:33969 (local address 127.8.221.193:44743)
0620 14:48:35.102124 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:35.102144 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:35.102295 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:35.102615 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:35.102618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:35.102851 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:35.102957 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.102964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.103897 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.103901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.104714 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.104726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.105372 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.105378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.105572 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.106294 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:35.106312 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:35.106820 (+ 508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:35.108921 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:35.108924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:35.108926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:35.109160 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:35.109504 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:35.109506 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:35.109508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:35.109588 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:35.110180 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:35.110182 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:35.110275 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 14:48:35.110447 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 14:48:35.110596 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":3,"thread_start_us":133,"threads_started":1}
I20260620 14:48:35.110980 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.101749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:44743 (local address 127.8.221.254:33969)
0620 14:48:35.101912 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.101918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.102222 (+ 304us) server_negotiation.cc:407] Connection header received
0620 14:48:35.102424 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.102428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.102479 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.102594 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:35.103095 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.103748 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.104838 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.105194 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.105329 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.106992 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:35.106995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:35.106997 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:35.107056 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:35.108772 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:35.109266 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:35.109269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:35.109270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:35.109357 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:35.109731 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:35.109734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:35.109735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:35.110043 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:35.110191 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:35.110533 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 14:48:35.110682 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":81}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.194.keytab.
Entry for principal oryx/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.194.keytab.
I20260620 14:48:35.128543 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33969
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.194.keytab
--principal=oryx/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:35.255002 16817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:35.255335 16817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:35.255430 16817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:35.259711 16817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:35.259773 16817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:35.259814 16817 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:48:35.259893 16817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:48:35.265726 16817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.194.keytab
--principal=oryx/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.221.254:33969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:35.267167 16817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:35.268505 16817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:35.280682 16825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:35.280793 16823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:35.280920 16817 server_base.cc:1034] running on GCE node
W20260620 14:48:35.280682 16822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:35.281582 16817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:35.282428 16817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:35.283630 16817 hybrid_clock.cc:648] HybridClock initialized: now 1781966915283610 us; error 43 us; skew 500 ppm
Jun 20 14:48:35 dist-test-slave-65tz krb5kdc[16560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966915, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:35.286862 16817 init.cc:377] Logged in from keytab as oryx/127.8.221.194@KRBTEST.COM (short username oryx)
I20260620 14:48:35.287904 16817 webserver.cc:460] Webserver started at http://127.8.221.194:33243/ using document root <none> and password file <none>
I20260620 14:48:35.288578 16817 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:35.288672 16817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:35.288931 16817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:35.291235 16817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "4a1f5d566aa440298ff26dc1610cde12"
format_stamp: "Formatted at 2026-06-20 14:48:35 on dist-test-slave-65tz"
server_key: "d50932af151a5e4bb8f346998c3e70f7"
server_key_iv: "0a9dfc0a6a7a5b416509c764c060a37d"
server_key_version: "encryptionkey@0"
I20260620 14:48:35.291831 16817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "4a1f5d566aa440298ff26dc1610cde12"
format_stamp: "Formatted at 2026-06-20 14:48:35 on dist-test-slave-65tz"
server_key: "d50932af151a5e4bb8f346998c3e70f7"
server_key_iv: "0a9dfc0a6a7a5b416509c764c060a37d"
server_key_version: "encryptionkey@0"
I20260620 14:48:35.296299 16817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:48:35.299612 16832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:35.301342 16817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:48:35.301491 16817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "4a1f5d566aa440298ff26dc1610cde12"
format_stamp: "Formatted at 2026-06-20 14:48:35 on dist-test-slave-65tz"
server_key: "d50932af151a5e4bb8f346998c3e70f7"
server_key_iv: "0a9dfc0a6a7a5b416509c764c060a37d"
server_key_version: "encryptionkey@0"
I20260620 14:48:35.301626 16817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:35.324890 16817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:35.328348 16817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:35.328579 16817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:35.330349 16817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:35.330426 16817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:35.330499 16817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:35.330543 16817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:48:35 dist-test-slave-65tz krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966915, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM
I20260620 14:48:35.342435 16817 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:43599
I20260620 14:48:35.342733 16947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:43599 every 8 connection(s)
I20260620 14:48:35.343662 16817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:35.346436 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.331593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:47755 (local address 127.8.221.254:33969)
0620 14:48:35.331752 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.331756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.332880 (+ 1124us) server_negotiation.cc:407] Connection header received
0620 14:48:35.334359 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.334365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.334438 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.334515 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:35.336381 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.337347 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.338961 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.339209 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.339326 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.342279 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:35.342282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:35.342285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:35.342310 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:35.344037 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:35.344764 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:35.344769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:35.344770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:35.344929 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:35.345297 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:35.345299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:35.345300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:35.345502 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:35.345588 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:35.346192 (+ 604us) server_negotiation.cc:299] Negotiation successful
0620 14:48:35.346316 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:48:35.346696 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 16817
I20260620 14:48:35.346834 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:35.347194 9079 external_mini_cluster.cc:1428] Setting key ff2318853f30746192d96cb3a6145add
I20260620 14:48:35.347651 16841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.332117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:33969 (local address 127.8.221.194:47755)
0620 14:48:35.332688 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:35.332743 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:35.334122 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:35.334709 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:35.334770 (+ 61us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:35.335400 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:35.336171 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.336188 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.337474 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.337478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.338811 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.338820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.339373 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.339378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.339576 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.340306 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:35.340324 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:35.342062 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:35.344177 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:35.344183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:35.344200 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:35.344574 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:35.345049 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:35.345051 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:35.345053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:35.345191 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:35.345597 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:35.345602 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:35.345941 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 14:48:35.346115 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 14:48:35.346451 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":152,"threads_started":1}
I20260620 14:48:35.349215 16948 heartbeater.cc:347] Connected to a master server at 127.8.221.254:33969
I20260620 14:48:35.349359 16948 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:35.350015 16948 heartbeater.cc:511] Master 127.8.221.254:33969 requested a full tablet report, sending...
I20260620 14:48:35.351261 16605 ts_manager.cc:194] Registered new tserver with Master: 4a1f5d566aa440298ff26dc1610cde12 (127.8.221.194:43599)
I20260620 14:48:35.352170 16605 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:47755
WARNING: no policy specified for oryx/127.8.221.195@KRBTEST.COM; defaulting to no policy
I20260620 14:48:35.360028 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.351850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:34527 (local address 127.8.221.254:33969)
0620 14:48:35.351995 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.352001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.352226 (+ 225us) server_negotiation.cc:407] Connection header received
0620 14:48:35.352382 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.352384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.352446 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.352542 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:35.353238 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.354072 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.355094 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.355332 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.355440 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.356638 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:35.356642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:35.356645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:35.356668 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:35.358337 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:35.358796 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:35.358800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:35.358802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:35.358880 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:35.359168 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:35.359171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:35.359172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:35.359388 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:35.359495 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:35.359767 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 14:48:35.359866 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:48:35.360042 16955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.351758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:33969 (local address 127.8.221.194:34527)
0620 14:48:35.352117 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:35.352132 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:35.352251 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:35.352679 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:35.352681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:35.352946 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:35.353098 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.353107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.354232 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.354235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.354975 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.354984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.355448 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.355451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.355549 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.356110 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:35.356122 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:35.356492 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:35.358448 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:35.358454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:35.358457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:35.358697 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:35.358974 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:35.358979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:35.358981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:35.359069 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:35.359495 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:35.359499 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:35.359592 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 14:48:35.359715 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 14:48:35.359866 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":2,"thread_start_us":128,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.221.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.195.keytab.
Entry for principal oryx/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.195.keytab.
I20260620 14:48:35.380877 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33969
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.195.keytab
--principal=oryx/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:35.513509 16957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:35.513784 16957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:35.513974 16957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:35.518076 16957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:35.518146 16957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:35.518189 16957 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:48:35.518227 16957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:48:35.523427 16957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.195.keytab
--principal=oryx/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.221.254:33969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:35.524768 16957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:35.526002 16957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:35.536504 16962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:35.536566 16963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:35.536566 16965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:35.536923 16957 server_base.cc:1034] running on GCE node
I20260620 14:48:35.537415 16957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:35.538100 16957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:35.539283 16957 hybrid_clock.cc:648] HybridClock initialized: now 1781966915539240 us; error 63 us; skew 500 ppm
Jun 20 14:48:35 dist-test-slave-65tz krb5kdc[16560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966915, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:35.542415 16957 init.cc:377] Logged in from keytab as oryx/127.8.221.195@KRBTEST.COM (short username oryx)
I20260620 14:48:35.543288 16957 webserver.cc:460] Webserver started at http://127.8.221.195:40925/ using document root <none> and password file <none>
I20260620 14:48:35.543888 16957 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:35.543985 16957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:35.544224 16957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:35.546401 16957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "c82552077c6e41a5b1c4bcb154b1ffa3"
format_stamp: "Formatted at 2026-06-20 14:48:35 on dist-test-slave-65tz"
server_key: "04d44fbd2edf47a22918d91f412ea6f6"
server_key_iv: "2a442b0108710dba5684ad0390258e95"
server_key_version: "encryptionkey@0"
I20260620 14:48:35.546996 16957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "c82552077c6e41a5b1c4bcb154b1ffa3"
format_stamp: "Formatted at 2026-06-20 14:48:35 on dist-test-slave-65tz"
server_key: "04d44fbd2edf47a22918d91f412ea6f6"
server_key_iv: "2a442b0108710dba5684ad0390258e95"
server_key_version: "encryptionkey@0"
I20260620 14:48:35.551237 16957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 14:48:35.554153 16972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:35.555650 16957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:48:35.555800 16957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "c82552077c6e41a5b1c4bcb154b1ffa3"
format_stamp: "Formatted at 2026-06-20 14:48:35 on dist-test-slave-65tz"
server_key: "04d44fbd2edf47a22918d91f412ea6f6"
server_key_iv: "2a442b0108710dba5684ad0390258e95"
server_key_version: "encryptionkey@0"
I20260620 14:48:35.555922 16957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:35.569054 16957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:35.572264 16957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:35.572505 16957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:35.574294 16957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:35.574370 16957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:35.574450 16957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:35.574499 16957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:48:35 dist-test-slave-65tz krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966915, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM
I20260620 14:48:35.587448 16957 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:38933
I20260620 14:48:35.587585 17087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:38933 every 8 connection(s)
I20260620 14:48:35.588619 16957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:35.589361 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 16957
I20260620 14:48:35.589505 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:35.589915 9079 external_mini_cluster.cc:1428] Setting key 2efe659704f56d880332f3356b048cdc
I20260620 14:48:35.591858 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.575424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:37729 (local address 127.8.221.254:33969)
0620 14:48:35.575629 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.575635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.576860 (+ 1225us) server_negotiation.cc:407] Connection header received
0620 14:48:35.578581 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.578589 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.578678 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.578830 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:35.580475 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.581113 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.582951 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.583197 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.583354 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.587093 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:35.587098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:35.587102 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:35.587139 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:35.588945 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:35.589713 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:35.589722 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:35.589724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:35.589915 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:35.590405 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:35.590410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:35.590412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:35.590714 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:35.590859 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:35.591571 (+ 712us) server_negotiation.cc:299] Negotiation successful
0620 14:48:35.591706 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 14:48:35.592833 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.575983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:33969 (local address 127.8.221.195:37729)
0620 14:48:35.576656 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:35.576725 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:35.578272 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:35.579070 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:35.579079 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:35.579680 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:35.580305 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.580322 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.581254 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.581258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.582793 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.582805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.583431 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.583436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.583639 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.584497 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:35.584516 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:35.586825 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:35.589085 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:35.589092 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:35.589113 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:35.589537 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:35.590064 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:35.590068 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:35.590070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:35.590275 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:35.590865 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:35.590876 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:35.591258 (+ 382us) client_negotiation.cc:769] Sending connection context
0620 14:48:35.591503 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 14:48:35.591864 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":176,"threads_started":1}
I20260620 14:48:35.594393 17088 heartbeater.cc:347] Connected to a master server at 127.8.221.254:33969
I20260620 14:48:35.594597 17088 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:35.595377 17088 heartbeater.cc:511] Master 127.8.221.254:33969 requested a full tablet report, sending...
I20260620 14:48:35.596722 16605 ts_manager.cc:194] Registered new tserver with Master: c82552077c6e41a5b1c4bcb154b1ffa3 (127.8.221.195:38933)
I20260620 14:48:35.597721 16605 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:37729
I20260620 14:48:35.605412 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.596775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:54533 (local address 127.8.221.254:33969)
0620 14:48:35.596920 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.596925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.597230 (+ 305us) server_negotiation.cc:407] Connection header received
0620 14:48:35.597395 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.597399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.597452 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.597555 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:35.598264 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.598903 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.600126 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.600360 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.600451 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.601678 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:35.601681 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:35.601684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:35.601707 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:35.603406 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:35.603904 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:35.603907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:35.603908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:35.603984 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:35.604275 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:35.604278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:35.604280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:35.604547 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:35.604671 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:35.605083 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 14:48:35.605256 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:48:35.605417 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.596658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:33969 (local address 127.8.221.195:54533)
0620 14:48:35.597098 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:35.597111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:35.597274 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:35.597599 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:35.597602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:35.597931 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:35.598127 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.598137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.599054 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.599057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.599991 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.600001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.600495 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.600497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.600578 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.601116 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:35.601139 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:35.601527 (+ 388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:35.603557 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:35.603560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:35.603563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:35.603797 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:35.604079 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:35.604082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:35.604083 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:35.604166 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:35.604677 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:35.604684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:35.604815 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 14:48:35.604940 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 14:48:35.605158 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":146,"threads_started":1}
I20260620 14:48:35.606405 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:48:35 dist-test-slave-65tz krb5kdc[16560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966915, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:48:35 dist-test-slave-65tz krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:35 dist-test-slave-65tz krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:48:35.634920 16654 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:35.627981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.8.221.254:33969)
0620 14:48:35.628164 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.628168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.628185 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:48:35.628233 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.628236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.628282 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.628366 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:35.629119 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.629681 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.630651 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.630875 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.630968 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.634796 (+ 3828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54402: BlockingRecv error: recv got EOF from 127.0.0.1:54402 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 14:48:35 dist-test-slave-65tz krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966915, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM
I20260620 14:48:35.645772 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.636919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54412 (local address 127.8.221.254:33969)
0620 14:48:35.637074 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.637078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.637141 (+ 63us) server_negotiation.cc:407] Connection header received
0620 14:48:35.637251 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.637254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.637301 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.637388 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:35.638106 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.638698 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.639561 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.639810 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.639902 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.642606 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:35.642610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:35.642613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:35.642642 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:35.644123 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:35.644588 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:35.644591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:35.644593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:35.644670 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:35.644965 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:35.644967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:35.644968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:35.645163 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:35.645272 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:35.645534 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 14:48:35.645647 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:48:35.649292 16605 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:48:35.652546 16605 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:35.669184 17112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.662028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:38933 (local address 127.0.0.1:59296)
0620 14:48:35.663312 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:35.663328 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:35.663649 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:35.664629 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:35.664632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:35.664645 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:35.664819 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.664827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.666192 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.666196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.667538 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.667550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.668602 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.668605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.668727 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.668768 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:48:35.668909 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 14:48:35.669029 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1226,"mutex_wait_us":57,"thread_start_us":175,"threads_started":1}
I20260620 14:48:35.669212 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.662092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.8.221.195:38933)
0620 14:48:35.662476 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.662492 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.663449 (+ 957us) server_negotiation.cc:407] Connection header received
0620 14:48:35.663932 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.663939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.664206 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.664417 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:35.664971 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.666055 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.667664 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.668468 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.668602 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.668772 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:35.668948 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 14:48:35.669050 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":127,"threads_started":1}
I20260620 14:48:35.669914 17110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.661857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:40715 (local address 127.0.0.1:51172)
0620 14:48:35.662579 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:35.662633 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:35.662846 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:35.663942 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:35.663950 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:35.663980 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:35.664187 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.664195 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.665641 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.665644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.667494 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.667505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.669417 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.669423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.669532 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.669550 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 14:48:35.669649 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:48:35.669733 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":69,"thread_start_us":87,"threads_started":1}
I20260620 14:48:35.670178 17114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.662807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:43599 (local address 127.0.0.1:54372)
0620 14:48:35.663877 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:35.663891 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:35.663971 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:35.664616 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:35.664619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:35.664639 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:35.664824 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.664829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.667100 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.667107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.668410 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.668422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.669694 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.669700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.669823 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.669860 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:48:35.669933 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:48:35.670028 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"mutex_wait_us":79,"spinlock_wait_cycles":2432,"thread_start_us":104,"threads_started":1}
I20260620 14:48:35.670315 17111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.662023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51172 (local address 127.8.221.193:40715)
0620 14:48:35.662653 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.662664 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.663045 (+ 381us) server_negotiation.cc:407] Connection header received
0620 14:48:35.663533 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.663543 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.663774 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.663931 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:35.664389 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.665515 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.667676 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.668418 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.668577 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.668775 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:35.669687 (+ 912us) server_negotiation.cc:299] Negotiation successful
0620 14:48:35.669847 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":120,"threads_started":1}
I20260620 14:48:35.670699 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.663046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54372 (local address 127.8.221.194:43599)
0620 14:48:35.663480 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.663498 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.664064 (+ 566us) server_negotiation.cc:407] Connection header received
0620 14:48:35.664175 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.664181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.664356 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.664612 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:35.666211 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.666964 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.668567 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.669314 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.669485 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.669630 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:35.670448 (+ 818us) server_negotiation.cc:299] Negotiation successful
0620 14:48:35.670541 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":128,"threads_started":1}
I20260620 14:48:35.673090 16742 tablet_service.cc:1467] Processing CreateTablet for tablet 52f34abfce8f40d4996c551845bd8cc3 (DEFAULT_TABLE table=test-table [id=aa4dddfbe43241ebbf21690e7a64bfa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:35.673460 16869 tablet_service.cc:1467] Processing CreateTablet for tablet 52f34abfce8f40d4996c551845bd8cc3 (DEFAULT_TABLE table=test-table [id=aa4dddfbe43241ebbf21690e7a64bfa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:35.673460 17022 tablet_service.cc:1467] Processing CreateTablet for tablet 52f34abfce8f40d4996c551845bd8cc3 (DEFAULT_TABLE table=test-table [id=aa4dddfbe43241ebbf21690e7a64bfa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:35.674211 16742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f34abfce8f40d4996c551845bd8cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:35.674738 17022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f34abfce8f40d4996c551845bd8cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:35.674738 16869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f34abfce8f40d4996c551845bd8cc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:35.680997 17116 tablet_bootstrap.cc:492] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269: Bootstrap starting.
I20260620 14:48:35.683578 17116 tablet_bootstrap.cc:654] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:35.684366 17117 tablet_bootstrap.cc:492] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3: Bootstrap starting.
I20260620 14:48:35.684617 17116 log.cc:824] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:35.685860 17118 tablet_bootstrap.cc:492] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12: Bootstrap starting.
I20260620 14:48:35.686915 17116 tablet_bootstrap.cc:492] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269: No bootstrap required, opened a new log
I20260620 14:48:35.687037 17117 tablet_bootstrap.cc:654] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:35.687121 17116 ts_tablet_manager.cc:1397] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:48:35.687939 17117 log.cc:824] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:35.688020 17118 tablet_bootstrap.cc:654] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:35.688799 17118 log.cc:824] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:35.690364 17117 tablet_bootstrap.cc:492] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3: No bootstrap required, opened a new log
I20260620 14:48:35.690595 17117 ts_tablet_manager.cc:1397] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:48:35.690711 17118 tablet_bootstrap.cc:492] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12: No bootstrap required, opened a new log
I20260620 14:48:35.690896 17118 ts_tablet_manager.cc:1397] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:48:35.691545 17116 raft_consensus.cc:348] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } }
I20260620 14:48:35.691816 17116 raft_consensus.cc:374] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:35.691866 17116 raft_consensus.cc:729] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4709532c534b7ab9d6911dda04a269, State: Initialized, Role: FOLLOWER
I20260620 14:48:35.692329 17116 consensus_queue.cc:260] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } }
I20260620 14:48:35.693270 17116 ts_tablet_manager.cc:1428] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:48:35.693707 16808 heartbeater.cc:503] Master 127.8.221.254:33969 was elected leader, sending a full tablet report...
I20260620 14:48:35.694577 17118 raft_consensus.cc:348] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } }
I20260620 14:48:35.694789 17118 raft_consensus.cc:374] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:35.694842 17118 raft_consensus.cc:729] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1f5d566aa440298ff26dc1610cde12, State: Initialized, Role: FOLLOWER
I20260620 14:48:35.694754 17117 raft_consensus.cc:348] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } }
I20260620 14:48:35.695063 17117 raft_consensus.cc:374] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:35.695123 17117 raft_consensus.cc:729] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82552077c6e41a5b1c4bcb154b1ffa3, State: Initialized, Role: FOLLOWER
I20260620 14:48:35.695250 17118 consensus_queue.cc:260] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } }
I20260620 14:48:35.695616 17117 consensus_queue.cc:260] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } }
I20260620 14:48:35.696161 17118 ts_tablet_manager.cc:1428] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:48:35.696484 16948 heartbeater.cc:503] Master 127.8.221.254:33969 was elected leader, sending a full tablet report...
I20260620 14:48:35.696904 17117 ts_tablet_manager.cc:1428] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:48:35.697402 17088 heartbeater.cc:503] Master 127.8.221.254:33969 was elected leader, sending a full tablet report...
I20260620 14:48:35.834394 17123 raft_consensus.cc:482] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:35.834746 17123 raft_consensus.cc:504] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } }
I20260620 14:48:35.836354 17123 leader_election.cc:288] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa4709532c534b7ab9d6911dda04a269 (127.8.221.193:40715), c82552077c6e41a5b1c4bcb154b1ffa3 (127.8.221.195:38933)
W20260620 14:48:35.839864 17089 tablet.cc:2369] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:48:35.841019 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.836844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:43975 (local address 127.8.221.195:38933)
0620 14:48:35.836994 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.837000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.837022 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:48:35.837117 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.837120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.837180 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.837307 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:35.837708 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.838467 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.839682 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.840379 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.840509 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.840590 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:35.840838 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 14:48:35.840902 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:48:35.841042 16841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.836727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:38933 (local address 127.8.221.194:43975)
0620 14:48:35.836859 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:35.836876 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:35.836981 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:35.837341 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:35.837347 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:35.837375 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:35.837580 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.837588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.838583 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.838587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.839521 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.839530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.840519 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.840522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.840677 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.840708 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:48:35.840799 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:48:35.840882 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":29,"spinlock_wait_cycles":24832}
I20260620 14:48:35.841820 17111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.836844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:47385 (local address 127.8.221.193:40715)
0620 14:48:35.836994 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.837000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.837430 (+ 430us) server_negotiation.cc:407] Connection header received
0620 14:48:35.837600 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.837603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.837665 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.837751 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:35.838105 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.838648 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.840613 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.841222 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.841359 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.841420 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:35.841613 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 14:48:35.841681 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:48:35.841850 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.836727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:40715 (local address 127.8.221.194:47385)
0620 14:48:35.837251 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:35.837270 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:35.837485 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:35.837778 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:35.837780 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:35.837838 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:35.837978 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.837985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.838765 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.838768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.840490 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.840499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.841369 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.841372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.841478 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.841507 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 14:48:35.841598 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:48:35.841677 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":195,"spinlock_wait_cycles":1920,"thread_start_us":102,"threads_started":1}
I20260620 14:48:35.841745 17042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f34abfce8f40d4996c551845bd8cc3" candidate_uuid: "4a1f5d566aa440298ff26dc1610cde12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" is_pre_election: true
I20260620 14:48:35.842145 17042 raft_consensus.cc:2454] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1f5d566aa440298ff26dc1610cde12 in term 0.
I20260620 14:48:35.842523 16762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f34abfce8f40d4996c551845bd8cc3" candidate_uuid: "4a1f5d566aa440298ff26dc1610cde12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4709532c534b7ab9d6911dda04a269" is_pre_election: true
I20260620 14:48:35.842721 16835 leader_election.cc:302] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1f5d566aa440298ff26dc1610cde12, c82552077c6e41a5b1c4bcb154b1ffa3; no voters:
I20260620 14:48:35.842854 16762 raft_consensus.cc:2454] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1f5d566aa440298ff26dc1610cde12 in term 0.
I20260620 14:48:35.843015 17123 raft_consensus.cc:2781] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:35.843127 17123 raft_consensus.cc:482] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:35.843189 17123 raft_consensus.cc:3037] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:35.844422 17123 raft_consensus.cc:504] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } }
I20260620 14:48:35.844835 17123 leader_election.cc:288] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [CANDIDATE]: Term 1 election: Requested vote from peers fa4709532c534b7ab9d6911dda04a269 (127.8.221.193:40715), c82552077c6e41a5b1c4bcb154b1ffa3 (127.8.221.195:38933)
W20260620 14:48:35.844923 16949 tablet.cc:2369] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:48:35.845249 16762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f34abfce8f40d4996c551845bd8cc3" candidate_uuid: "4a1f5d566aa440298ff26dc1610cde12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4709532c534b7ab9d6911dda04a269"
I20260620 14:48:35.845263 17042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f34abfce8f40d4996c551845bd8cc3" candidate_uuid: "4a1f5d566aa440298ff26dc1610cde12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3"
I20260620 14:48:35.845386 16762 raft_consensus.cc:3037] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:35.845408 17042 raft_consensus.cc:3037] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Advancing to term 1
W20260620 14:48:35.845610 16809 tablet.cc:2369] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:48:35.846663 16762 raft_consensus.cc:2454] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1f5d566aa440298ff26dc1610cde12 in term 1.
I20260620 14:48:35.847018 16833 leader_election.cc:302] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1f5d566aa440298ff26dc1610cde12, fa4709532c534b7ab9d6911dda04a269; no voters:
I20260620 14:48:35.847222 17123 raft_consensus.cc:2781] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:35.847295 17042 raft_consensus.cc:2454] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1f5d566aa440298ff26dc1610cde12 in term 1.
I20260620 14:48:35.847519 17123 raft_consensus.cc:686] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 LEADER]: Becoming Leader. State: Replica: 4a1f5d566aa440298ff26dc1610cde12, State: Running, Role: LEADER
I20260620 14:48:35.847951 17123 consensus_queue.cc:237] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } }
I20260620 14:48:35.851610 16605 catalog_manager.cc:5495] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1f5d566aa440298ff26dc1610cde12 (127.8.221.194). New cstate: current_term: 1 leader_uuid: "4a1f5d566aa440298ff26dc1610cde12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } health_report { overall_health: HEALTHY } } }
I20260620 14:48:35.867075 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.863195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.8.221.194:43599)
0620 14:48:35.863356 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.863360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.863371 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:48:35.863438 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.863440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.863539 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.863695 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:35.864202 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.864722 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.865582 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.865860 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.865970 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.866247 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:35.866679 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:35.866799 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:35.866873 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 14:48:35.866924 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:48:35.874043 17042 raft_consensus.cc:1261] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 1 FOLLOWER]: Refusing update from remote peer 4a1f5d566aa440298ff26dc1610cde12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:35.874253 16762 raft_consensus.cc:1261] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269 [term 1 FOLLOWER]: Refusing update from remote peer 4a1f5d566aa440298ff26dc1610cde12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:35.874814 17123 consensus_queue.cc:1035] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:35.875083 17126 consensus_queue.cc:1035] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:35.887509 17131 mvcc.cc:205] Tried to move back new op lower bound from 7298936487410900992 to 7298936487316299776. Current Snapshot: MvccSnapshot[applied={T|T < 7298936487410900992}]
I20260620 14:48:35.894083 16600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:35.894285 16600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:35.897445 16600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885: Sending DeleteTablet for 3 replicas of tablet 52f34abfce8f40d4996c551845bd8cc3
I20260620 14:48:35.898391 16742 tablet_service.cc:1514] Processing DeleteTablet for tablet 52f34abfce8f40d4996c551845bd8cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:35 UTC) from {username='oryx', principal='oryx/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:51172
I20260620 14:48:35.898515 17022 tablet_service.cc:1514] Processing DeleteTablet for tablet 52f34abfce8f40d4996c551845bd8cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:35 UTC) from {username='oryx', principal='oryx/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:59296
I20260620 14:48:35.898912 17138 tablet_replica.cc:330] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269: stopping tablet replica
I20260620 14:48:35.898878 16869 tablet_service.cc:1514] Processing DeleteTablet for tablet 52f34abfce8f40d4996c551845bd8cc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:35 UTC) from {username='oryx', principal='oryx/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:54372
I20260620 14:48:35.899392 17140 tablet_replica.cc:330] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12: stopping tablet replica
I20260620 14:48:35.899398 17138 raft_consensus.cc:2229] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:35.899737 17140 raft_consensus.cc:2229] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:35.899888 17138 raft_consensus.cc:2258] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:35.900120 16600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:48:35.900256 17140 raft_consensus.cc:2258] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 14:48:35.900655 16600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:35.902051 17140 ts_tablet_manager.cc:1905] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:35.902647 17139 tablet_replica.cc:330] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3: stopping tablet replica
I20260620 14:48:35.902989 17138 ts_tablet_manager.cc:1905] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:35.903000 17139 raft_consensus.cc:2229] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:35.903352 17139 raft_consensus.cc:2258] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:35.905125 17139 ts_tablet_manager.cc:1905] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:35.906152 17138 ts_tablet_manager.cc:1918] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:35.906292 17138 log.cc:1196] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/wal/wals/52f34abfce8f40d4996c551845bd8cc3
I20260620 14:48:35.906697 17138 ts_tablet_manager.cc:1939] T 52f34abfce8f40d4996c551845bd8cc3 P fa4709532c534b7ab9d6911dda04a269: Deleting consensus metadata
I20260620 14:48:35.908255 16742 tablet_service.cc:1467] Processing CreateTablet for tablet 56ee3fc6bd294119868f7a26d119ab43 (DEFAULT_TABLE table=test-table [id=d875e2f4716043658f7d52de80577b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:35.908262 17022 tablet_service.cc:1467] Processing CreateTablet for tablet 56ee3fc6bd294119868f7a26d119ab43 (DEFAULT_TABLE table=test-table [id=d875e2f4716043658f7d52de80577b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:35.908517 16742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ee3fc6bd294119868f7a26d119ab43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:35.908553 17022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ee3fc6bd294119868f7a26d119ab43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:35.908540 16869 tablet_service.cc:1467] Processing CreateTablet for tablet 56ee3fc6bd294119868f7a26d119ab43 (DEFAULT_TABLE table=test-table [id=d875e2f4716043658f7d52de80577b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:35.908823 16869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ee3fc6bd294119868f7a26d119ab43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:35.911254 17116 tablet_bootstrap.cc:492] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269: Bootstrap starting.
I20260620 14:48:35.911593 17117 tablet_bootstrap.cc:492] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3: Bootstrap starting.
I20260620 14:48:35.912258 17116 tablet_bootstrap.cc:654] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:35.912400 17139 ts_tablet_manager.cc:1918] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:35.912501 17139 log.cc:1196] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/52f34abfce8f40d4996c551845bd8cc3
I20260620 14:48:35.912792 17117 tablet_bootstrap.cc:654] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:35.912851 17139 ts_tablet_manager.cc:1939] T 52f34abfce8f40d4996c551845bd8cc3 P c82552077c6e41a5b1c4bcb154b1ffa3: Deleting consensus metadata
I20260620 14:48:35.913508 16589 catalog_manager.cc:4841] TS fa4709532c534b7ab9d6911dda04a269 (127.8.221.193:40715): tablet 52f34abfce8f40d4996c551845bd8cc3 (table test-table [id=aa4dddfbe43241ebbf21690e7a64bfa3]) successfully deleted
I20260620 14:48:35.914399 16591 catalog_manager.cc:4841] TS c82552077c6e41a5b1c4bcb154b1ffa3 (127.8.221.195:38933): tablet 52f34abfce8f40d4996c551845bd8cc3 (table test-table [id=aa4dddfbe43241ebbf21690e7a64bfa3]) successfully deleted
I20260620 14:48:35.914430 17117 tablet_bootstrap.cc:492] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3: No bootstrap required, opened a new log
I20260620 14:48:35.914537 17117 ts_tablet_manager.cc:1397] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:35.914798 17118 tablet_bootstrap.cc:492] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12: Bootstrap starting.
I20260620 14:48:35.915058 17117 raft_consensus.cc:348] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:35.915186 17117 raft_consensus.cc:374] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:35.915216 17117 raft_consensus.cc:729] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82552077c6e41a5b1c4bcb154b1ffa3, State: Initialized, Role: FOLLOWER
I20260620 14:48:35.915352 17117 consensus_queue.cc:260] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:35.915619 17117 ts_tablet_manager.cc:1428] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:48:35.915882 17118 tablet_bootstrap.cc:654] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:35.917415 17118 tablet_bootstrap.cc:492] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12: No bootstrap required, opened a new log
I20260620 14:48:35.917511 17118 ts_tablet_manager.cc:1397] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:35.918102 17118 raft_consensus.cc:348] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:35.918241 17118 raft_consensus.cc:374] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:35.918305 17118 raft_consensus.cc:729] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1f5d566aa440298ff26dc1610cde12, State: Initialized, Role: FOLLOWER
I20260620 14:48:35.918438 17118 consensus_queue.cc:260] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:35.919431 17118 ts_tablet_manager.cc:1428] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:48:35.922323 17140 ts_tablet_manager.cc:1918] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:35.922439 17140 log.cc:1196] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/52f34abfce8f40d4996c551845bd8cc3
I20260620 14:48:35.922797 17140 ts_tablet_manager.cc:1939] T 52f34abfce8f40d4996c551845bd8cc3 P 4a1f5d566aa440298ff26dc1610cde12: Deleting consensus metadata
I20260620 14:48:35.924026 16591 catalog_manager.cc:4841] TS 4a1f5d566aa440298ff26dc1610cde12 (127.8.221.194:43599): tablet 52f34abfce8f40d4996c551845bd8cc3 (table test-table [id=aa4dddfbe43241ebbf21690e7a64bfa3]) successfully deleted
I20260620 14:48:35.924902 17116 tablet_bootstrap.cc:492] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269: No bootstrap required, opened a new log
I20260620 14:48:35.925022 17116 ts_tablet_manager.cc:1397] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260620 14:48:35.925694 17116 raft_consensus.cc:348] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:35.925859 17116 raft_consensus.cc:374] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:35.925904 17116 raft_consensus.cc:729] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4709532c534b7ab9d6911dda04a269, State: Initialized, Role: FOLLOWER
I20260620 14:48:35.926031 17116 consensus_queue.cc:260] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:35.926299 17116 ts_tablet_manager.cc:1428] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 14:48:35.930554 17122 raft_consensus.cc:482] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:35.930673 17122 raft_consensus.cc:504] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:35.931926 17122 leader_election.cc:288] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1f5d566aa440298ff26dc1610cde12 (127.8.221.194:43599), c82552077c6e41a5b1c4bcb154b1ffa3 (127.8.221.195:38933)
I20260620 14:48:35.938095 17113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.932919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:34997 (local address 127.8.221.195:38933)
0620 14:48:35.933040 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.933044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.933060 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:48:35.933117 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.933120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.933173 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.933267 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:35.933624 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.934435 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.936571 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.937206 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.937332 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.937384 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:35.937701 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 14:48:35.937922 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:48:35.938195 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.932199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:43599 (local address 127.8.221.193:33377)
0620 14:48:35.932475 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:35.932492 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:35.932649 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:35.933388 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:35.933391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:35.933403 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:35.933517 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.933521 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.935489 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.935493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.936792 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.936799 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.937637 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.937641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.937773 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.937833 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 14:48:35.937935 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 14:48:35.938035 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
I20260620 14:48:35.938206 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.932291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:33377 (local address 127.8.221.194:43599)
0620 14:48:35.932450 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.932455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.932585 (+ 130us) server_negotiation.cc:407] Connection header received
0620 14:48:35.932809 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.932812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.932882 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.932964 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:35.934609 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.935377 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.936921 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.937487 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.937635 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.937700 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:35.937978 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 14:48:35.938064 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:48:35.938843 16902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ee3fc6bd294119868f7a26d119ab43" candidate_uuid: "fa4709532c534b7ab9d6911dda04a269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1f5d566aa440298ff26dc1610cde12" is_pre_election: true
I20260620 14:48:35.938982 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.932525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:38933 (local address 127.8.221.193:34997)
0620 14:48:35.932881 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:35.932895 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:35.932981 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:35.933296 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:35.933299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:35.933318 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:35.933500 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.933508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.934602 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.934606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.936426 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.936438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.937369 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.937373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.937498 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.937544 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 14:48:35.938787 (+ 1243us) client_negotiation.cc:240] Negotiation successful
0620 14:48:35.938861 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":105,"threads_started":1}
I20260620 14:48:35.939119 16902 raft_consensus.cc:2454] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa4709532c534b7ab9d6911dda04a269 in term 0.
I20260620 14:48:35.939618 16695 leader_election.cc:302] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1f5d566aa440298ff26dc1610cde12, fa4709532c534b7ab9d6911dda04a269; no voters:
I20260620 14:48:35.939776 17042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ee3fc6bd294119868f7a26d119ab43" candidate_uuid: "fa4709532c534b7ab9d6911dda04a269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" is_pre_election: true
I20260620 14:48:35.939903 17122 raft_consensus.cc:2781] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:35.939965 17122 raft_consensus.cc:482] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:35.939950 17042 raft_consensus.cc:2454] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa4709532c534b7ab9d6911dda04a269 in term 0.
I20260620 14:48:35.940011 17122 raft_consensus.cc:3037] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:35.941555 17122 raft_consensus.cc:504] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:35.941977 17122 leader_election.cc:288] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [CANDIDATE]: Term 1 election: Requested vote from peers 4a1f5d566aa440298ff26dc1610cde12 (127.8.221.194:43599), c82552077c6e41a5b1c4bcb154b1ffa3 (127.8.221.195:38933)
I20260620 14:48:35.942386 16902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ee3fc6bd294119868f7a26d119ab43" candidate_uuid: "fa4709532c534b7ab9d6911dda04a269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1f5d566aa440298ff26dc1610cde12"
I20260620 14:48:35.942435 17042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ee3fc6bd294119868f7a26d119ab43" candidate_uuid: "fa4709532c534b7ab9d6911dda04a269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3"
I20260620 14:48:35.942512 16902 raft_consensus.cc:3037] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:35.942601 17042 raft_consensus.cc:3037] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:35.943651 17042 raft_consensus.cc:2454] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa4709532c534b7ab9d6911dda04a269 in term 1.
I20260620 14:48:35.943868 16902 raft_consensus.cc:2454] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa4709532c534b7ab9d6911dda04a269 in term 1.
I20260620 14:48:35.944067 16695 leader_election.cc:302] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c82552077c6e41a5b1c4bcb154b1ffa3, fa4709532c534b7ab9d6911dda04a269; no voters:
I20260620 14:48:35.944268 17122 raft_consensus.cc:2781] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:35.944557 17122 raft_consensus.cc:686] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [term 1 LEADER]: Becoming Leader. State: Replica: fa4709532c534b7ab9d6911dda04a269, State: Running, Role: LEADER
I20260620 14:48:35.944871 17122 consensus_queue.cc:237] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:35.947908 16600 catalog_manager.cc:5495] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa4709532c534b7ab9d6911dda04a269 (127.8.221.193). New cstate: current_term: 1 leader_uuid: "fa4709532c534b7ab9d6911dda04a269" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:48:35 dist-test-slave-65tz krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966914, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.254@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM
I20260620 14:48:35.986835 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.976188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:33969 (local address 127.0.0.1:54418)
0620 14:48:35.976550 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:35.976577 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:35.976705 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:35.977079 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:35.977082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:35.977575 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:35.977700 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.977707 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.978893 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.978896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.979577 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.979585 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.980180 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.980183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.980260 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.980978 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:35.980990 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:35.982806 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:35.984920 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:35.984929 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:35.984935 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:35.985222 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:35.985575 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:35.985578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:35.985579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:35.985674 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:35.986179 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:35.986186 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:35.986374 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 14:48:35.986493 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 14:48:35.986645 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":137,"threads_started":1}
I20260620 14:48:35.986945 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.976323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.8.221.254:33969)
0620 14:48:35.976492 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.976497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.976675 (+ 178us) server_negotiation.cc:407] Connection header received
0620 14:48:35.976844 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.976848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.976911 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.977034 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:35.977909 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.978749 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.979705 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.980044 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.980156 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.982961 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:35.982965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:35.982969 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:35.982997 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:35.984780 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:35.985338 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:35.985341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:35.985343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:35.985434 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:35.985779 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:35.985782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:35.985784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:35.986041 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:35.986149 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:35.986590 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 14:48:35.986717 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:48:35.992493 16600 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:54418:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:48:35.993468 16600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:35.999528 16742 tablet_service.cc:1467] Processing CreateTablet for tablet d5d3d9d854924433899fd951550b84a5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=13c169de4f344276bdb23024feacd46b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:48:35.999800 17022 tablet_service.cc:1467] Processing CreateTablet for tablet d5d3d9d854924433899fd951550b84a5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=13c169de4f344276bdb23024feacd46b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:48:35.999769 16869 tablet_service.cc:1467] Processing CreateTablet for tablet d5d3d9d854924433899fd951550b84a5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=13c169de4f344276bdb23024feacd46b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:48:35.999976 16742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d3d9d854924433899fd951550b84a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:36.000217 16869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d3d9d854924433899fd951550b84a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:36.000217 17022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5d3d9d854924433899fd951550b84a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:36.003418 17118 tablet_bootstrap.cc:492] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12: Bootstrap starting.
I20260620 14:48:36.003528 17117 tablet_bootstrap.cc:492] T d5d3d9d854924433899fd951550b84a5 P c82552077c6e41a5b1c4bcb154b1ffa3: Bootstrap starting.
I20260620 14:48:36.004506 17118 tablet_bootstrap.cc:654] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:36.004693 17117 tablet_bootstrap.cc:654] T d5d3d9d854924433899fd951550b84a5 P c82552077c6e41a5b1c4bcb154b1ffa3: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:36.005453 17116 tablet_bootstrap.cc:492] T d5d3d9d854924433899fd951550b84a5 P fa4709532c534b7ab9d6911dda04a269: Bootstrap starting.
I20260620 14:48:36.006676 17116 tablet_bootstrap.cc:654] T d5d3d9d854924433899fd951550b84a5 P fa4709532c534b7ab9d6911dda04a269: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:36.007018 17118 tablet_bootstrap.cc:492] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12: No bootstrap required, opened a new log
I20260620 14:48:36.007136 17118 ts_tablet_manager.cc:1397] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:36.007670 17118 raft_consensus.cc:348] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.007817 17118 raft_consensus.cc:374] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:36.007851 17118 raft_consensus.cc:729] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1f5d566aa440298ff26dc1610cde12, State: Initialized, Role: FOLLOWER
I20260620 14:48:36.007999 17118 consensus_queue.cc:260] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.008082 17116 tablet_bootstrap.cc:492] T d5d3d9d854924433899fd951550b84a5 P fa4709532c534b7ab9d6911dda04a269: No bootstrap required, opened a new log
I20260620 14:48:36.008199 17116 ts_tablet_manager.cc:1397] T d5d3d9d854924433899fd951550b84a5 P fa4709532c534b7ab9d6911dda04a269: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:36.008306 17118 ts_tablet_manager.cc:1428] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:36.008512 17126 tablet_replica.cc:439] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } } }
I20260620 14:48:36.008666 17126 tablet_replica.cc:442] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:36.008708 17116 raft_consensus.cc:348] T d5d3d9d854924433899fd951550b84a5 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.008862 17116 raft_consensus.cc:374] T d5d3d9d854924433899fd951550b84a5 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:36.008903 17116 raft_consensus.cc:729] T d5d3d9d854924433899fd951550b84a5 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4709532c534b7ab9d6911dda04a269, State: Initialized, Role: FOLLOWER
I20260620 14:48:36.009042 17116 consensus_queue.cc:260] T d5d3d9d854924433899fd951550b84a5 P fa4709532c534b7ab9d6911dda04a269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.009341 17116 ts_tablet_manager.cc:1428] T d5d3d9d854924433899fd951550b84a5 P fa4709532c534b7ab9d6911dda04a269: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 14:48:36.009272 17122 tablet_replica.cc:439] T d5d3d9d854924433899fd951550b84a5 P fa4709532c534b7ab9d6911dda04a269: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } } }
I20260620 14:48:36.009419 17122 tablet_replica.cc:442] T d5d3d9d854924433899fd951550b84a5 P fa4709532c534b7ab9d6911dda04a269: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:36.010586 17117 tablet_bootstrap.cc:492] T d5d3d9d854924433899fd951550b84a5 P c82552077c6e41a5b1c4bcb154b1ffa3: No bootstrap required, opened a new log
I20260620 14:48:36.010682 17117 ts_tablet_manager.cc:1397] T d5d3d9d854924433899fd951550b84a5 P c82552077c6e41a5b1c4bcb154b1ffa3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 14:48:36.011215 17117 raft_consensus.cc:348] T d5d3d9d854924433899fd951550b84a5 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.011356 17117 raft_consensus.cc:374] T d5d3d9d854924433899fd951550b84a5 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:36.011394 17117 raft_consensus.cc:729] T d5d3d9d854924433899fd951550b84a5 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82552077c6e41a5b1c4bcb154b1ffa3, State: Initialized, Role: FOLLOWER
I20260620 14:48:36.011538 17117 consensus_queue.cc:260] T d5d3d9d854924433899fd951550b84a5 P c82552077c6e41a5b1c4bcb154b1ffa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.011821 17117 ts_tablet_manager.cc:1428] T d5d3d9d854924433899fd951550b84a5 P c82552077c6e41a5b1c4bcb154b1ffa3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:48:36.011842 17124 tablet_replica.cc:439] T d5d3d9d854924433899fd951550b84a5 P c82552077c6e41a5b1c4bcb154b1ffa3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } } }
I20260620 14:48:36.012002 17124 tablet_replica.cc:442] T d5d3d9d854924433899fd951550b84a5 P c82552077c6e41a5b1c4bcb154b1ffa3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:36.269135 17126 raft_consensus.cc:482] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:36.269295 17126 raft_consensus.cc:504] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.269928 17126 leader_election.cc:288] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa4709532c534b7ab9d6911dda04a269 (127.8.221.193:40715), c82552077c6e41a5b1c4bcb154b1ffa3 (127.8.221.195:38933)
I20260620 14:48:36.270594 16762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d3d9d854924433899fd951550b84a5" candidate_uuid: "4a1f5d566aa440298ff26dc1610cde12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4709532c534b7ab9d6911dda04a269" is_pre_election: true
I20260620 14:48:36.270594 17042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d3d9d854924433899fd951550b84a5" candidate_uuid: "4a1f5d566aa440298ff26dc1610cde12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" is_pre_election: true
I20260620 14:48:36.270817 16762 raft_consensus.cc:2454] T d5d3d9d854924433899fd951550b84a5 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1f5d566aa440298ff26dc1610cde12 in term 0.
I20260620 14:48:36.270839 17042 raft_consensus.cc:2454] T d5d3d9d854924433899fd951550b84a5 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1f5d566aa440298ff26dc1610cde12 in term 0.
I20260620 14:48:36.271210 16835 leader_election.cc:302] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1f5d566aa440298ff26dc1610cde12, c82552077c6e41a5b1c4bcb154b1ffa3; no voters:
I20260620 14:48:36.271433 17126 raft_consensus.cc:2781] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:36.271507 17126 raft_consensus.cc:482] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:36.271544 17126 raft_consensus.cc:3037] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:36.272931 17126 raft_consensus.cc:504] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.273358 17126 leader_election.cc:288] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 [CANDIDATE]: Term 1 election: Requested vote from peers fa4709532c534b7ab9d6911dda04a269 (127.8.221.193:40715), c82552077c6e41a5b1c4bcb154b1ffa3 (127.8.221.195:38933)
I20260620 14:48:36.273847 17042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d3d9d854924433899fd951550b84a5" candidate_uuid: "4a1f5d566aa440298ff26dc1610cde12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3"
I20260620 14:48:36.273986 17042 raft_consensus.cc:3037] T d5d3d9d854924433899fd951550b84a5 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:36.273967 16762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5d3d9d854924433899fd951550b84a5" candidate_uuid: "4a1f5d566aa440298ff26dc1610cde12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4709532c534b7ab9d6911dda04a269"
I20260620 14:48:36.274164 16762 raft_consensus.cc:3037] T d5d3d9d854924433899fd951550b84a5 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:36.275120 17042 raft_consensus.cc:2454] T d5d3d9d854924433899fd951550b84a5 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1f5d566aa440298ff26dc1610cde12 in term 1.
I20260620 14:48:36.275348 16762 raft_consensus.cc:2454] T d5d3d9d854924433899fd951550b84a5 P fa4709532c534b7ab9d6911dda04a269 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1f5d566aa440298ff26dc1610cde12 in term 1.
I20260620 14:48:36.275439 16835 leader_election.cc:302] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1f5d566aa440298ff26dc1610cde12, c82552077c6e41a5b1c4bcb154b1ffa3; no voters:
I20260620 14:48:36.275691 17126 raft_consensus.cc:2781] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:36.275854 17126 raft_consensus.cc:686] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 LEADER]: Becoming Leader. State: Replica: 4a1f5d566aa440298ff26dc1610cde12, State: Running, Role: LEADER
I20260620 14:48:36.276019 17126 consensus_queue.cc:237] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.276742 17123 tablet_replica.cc:439] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12: TxnStatusTablet state changed. Reason: New leader 4a1f5d566aa440298ff26dc1610cde12. Latest consensus state: current_term: 1 leader_uuid: "4a1f5d566aa440298ff26dc1610cde12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } } }
I20260620 14:48:36.276872 17123 tablet_replica.cc:442] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:48:36.277184 17158 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:48:36.277375 17158 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:48:36.278146 16603 catalog_manager.cc:5495] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1f5d566aa440298ff26dc1610cde12 (127.8.221.194). New cstate: current_term: 1 leader_uuid: "4a1f5d566aa440298ff26dc1610cde12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } health_report { overall_health: UNKNOWN } } }
I20260620 14:48:36.315995 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:36.311930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:43599 (local address 127.0.0.1:54400)
0620 14:48:36.312090 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:36.312106 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:36.312217 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:36.312584 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:36.312586 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:36.312595 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:48:36.312709 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:36.312714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.313873 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.313877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:36.314635 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:36.314645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.315260 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.315263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:36.315335 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:36.315394 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:48:36.315734 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:48:36.315746 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 14:48:36.315805 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:48:36.315872 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 14:48:36.316058 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:36.312100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.8.221.194:43599)
0620 14:48:36.312244 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:36.312249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:36.312269 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:48:36.312337 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:36.312340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:36.312399 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:36.312497 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:36.312841 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.313666 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.314764 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.315118 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.315235 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:36.315549 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:36.315642 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:36.315719 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:36.315827 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 14:48:36.315890 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:48:36.319165 17042 raft_consensus.cc:1261] T d5d3d9d854924433899fd951550b84a5 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 1 FOLLOWER]: Refusing update from remote peer 4a1f5d566aa440298ff26dc1610cde12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:36.319235 16762 raft_consensus.cc:1261] T d5d3d9d854924433899fd951550b84a5 P fa4709532c534b7ab9d6911dda04a269 [term 1 FOLLOWER]: Refusing update from remote peer 4a1f5d566aa440298ff26dc1610cde12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:36.319743 17123 consensus_queue.cc:1035] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:36.319981 17126 consensus_queue.cc:1035] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:36.321661 17124 tablet_replica.cc:439] T d5d3d9d854924433899fd951550b84a5 P c82552077c6e41a5b1c4bcb154b1ffa3: TxnStatusTablet state changed. Reason: New leader 4a1f5d566aa440298ff26dc1610cde12. Latest consensus state: current_term: 1 leader_uuid: "4a1f5d566aa440298ff26dc1610cde12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } } }
I20260620 14:48:36.321831 17124 tablet_replica.cc:442] T d5d3d9d854924433899fd951550b84a5 P c82552077c6e41a5b1c4bcb154b1ffa3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:36.322669 17122 tablet_replica.cc:439] T d5d3d9d854924433899fd951550b84a5 P fa4709532c534b7ab9d6911dda04a269: TxnStatusTablet state changed. Reason: New leader 4a1f5d566aa440298ff26dc1610cde12. Latest consensus state: current_term: 1 leader_uuid: "4a1f5d566aa440298ff26dc1610cde12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } } }
I20260620 14:48:36.322813 17122 tablet_replica.cc:442] T d5d3d9d854924433899fd951550b84a5 P fa4709532c534b7ab9d6911dda04a269: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:36.322795 17123 tablet_replica.cc:439] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a1f5d566aa440298ff26dc1610cde12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } } }
I20260620 14:48:36.322924 17123 tablet_replica.cc:442] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:48:36.325163 17124 tablet_replica.cc:439] T d5d3d9d854924433899fd951550b84a5 P c82552077c6e41a5b1c4bcb154b1ffa3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a1f5d566aa440298ff26dc1610cde12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } } }
I20260620 14:48:36.325292 17124 tablet_replica.cc:442] T d5d3d9d854924433899fd951550b84a5 P c82552077c6e41a5b1c4bcb154b1ffa3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:36.325942 17129 mvcc.cc:205] Tried to move back new op lower bound from 7298936489238495232 to 7298936489067995136. Current Snapshot: MvccSnapshot[applied={T|T < 7298936489238495232 or (T in {7298936489238495232})}]
I20260620 14:48:36.327491 17126 tablet_replica.cc:439] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a1f5d566aa440298ff26dc1610cde12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } } }
I20260620 14:48:36.327600 17126 tablet_replica.cc:442] T d5d3d9d854924433899fd951550b84a5 P 4a1f5d566aa440298ff26dc1610cde12: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:48:36.328375 17130 mvcc.cc:205] Tried to move back new op lower bound from 7298936489238495232 to 7298936489067995136. Current Snapshot: MvccSnapshot[applied={T|T < 7298936489238495232}]
I20260620 14:48:36.330256 17122 tablet_replica.cc:439] T d5d3d9d854924433899fd951550b84a5 P fa4709532c534b7ab9d6911dda04a269: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a1f5d566aa440298ff26dc1610cde12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } } }
I20260620 14:48:36.330376 17122 tablet_replica.cc:442] T d5d3d9d854924433899fd951550b84a5 P fa4709532c534b7ab9d6911dda04a269: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:36.332854 17111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:36.328060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51182 (local address 127.8.221.193:40715)
0620 14:48:36.328169 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:36.328174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:36.328190 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:48:36.328372 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:36.328374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:36.328438 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:36.328567 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:36.329192 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.330041 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.331203 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.331559 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.331660 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:36.332042 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:36.332486 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:36.332583 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:36.332705 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 14:48:36.332755 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:48:36.345351 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:36.340866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:43599 (local address 127.8.221.193:43693)
0620 14:48:36.341258 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:36.341276 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:36.341402 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:36.341717 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:36.341719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:36.341725 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:48:36.341896 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:36.341902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.342924 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.342927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:36.343920 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:36.343930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.344533 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.344537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:36.344635 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:36.344683 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:48:36.345020 (+ 337us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:48:36.345025 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:48:36.345102 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:48:36.345176 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":119,"threads_started":1}
I20260620 14:48:36.345356 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:36.340984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:43693 (local address 127.8.221.194:43599)
0620 14:48:36.341122 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:36.341127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:36.341364 (+ 237us) server_negotiation.cc:407] Connection header received
0620 14:48:36.341539 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:36.341543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:36.341604 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:36.341696 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:36.342005 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.342799 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.344052 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.344417 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.344519 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:36.344833 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:36.344927 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:36.345001 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:36.345132 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 14:48:36.345190 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:48:36.353348 17042 raft_consensus.cc:1261] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 1 FOLLOWER]: Refusing update from remote peer fa4709532c534b7ab9d6911dda04a269: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:36.353616 16902 raft_consensus.cc:1261] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 FOLLOWER]: Refusing update from remote peer fa4709532c534b7ab9d6911dda04a269: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:36.353853 17122 consensus_queue.cc:1035] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:36.354140 17146 consensus_queue.cc:1035] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:36.383522 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:36.378680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:40715 (local address 127.8.221.194:51437)
0620 14:48:36.379115 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:36.379131 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:36.379253 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:36.379643 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:36.379646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:36.379653 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:48:36.379767 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:36.379772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.380903 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.380906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:36.381925 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:36.381934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.382488 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.382492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:36.382629 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:36.382703 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:48:36.383096 (+ 393us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:48:36.383103 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:48:36.383266 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 14:48:36.383381 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":133,"threads_started":1}
I20260620 14:48:36.383533 17111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:36.378764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:51437 (local address 127.8.221.193:40715)
0620 14:48:36.378902 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:36.378906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:36.379227 (+ 321us) server_negotiation.cc:407] Connection header received
0620 14:48:36.379391 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:36.379394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:36.379455 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:36.379502 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:36.379924 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.380764 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.382080 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.382352 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.382473 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:36.382877 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:36.382971 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:36.383057 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:36.383289 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 14:48:36.383377 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:48:36.412375 16600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:36.412556 16600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:36.414592 16600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885: Sending DeleteTablet for 3 replicas of tablet 56ee3fc6bd294119868f7a26d119ab43
I20260620 14:48:36.415263 16742 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ee3fc6bd294119868f7a26d119ab43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:36 UTC) from {username='oryx', principal='oryx/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:51172
I20260620 14:48:36.415303 17022 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ee3fc6bd294119868f7a26d119ab43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:36 UTC) from {username='oryx', principal='oryx/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:59296
I20260620 14:48:36.415719 16869 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ee3fc6bd294119868f7a26d119ab43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:36 UTC) from {username='oryx', principal='oryx/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:54372
I20260620 14:48:36.415951 17140 tablet_replica.cc:330] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12: stopping tablet replica
I20260620 14:48:36.416182 17140 raft_consensus.cc:2229] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:36.416201 17174 tablet_replica.cc:330] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269: stopping tablet replica
I20260620 14:48:36.416338 17140 raft_consensus.cc:2258] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:36.416342 17174 raft_consensus.cc:2229] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:36.416692 17174 raft_consensus.cc:2258] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:36.416929 17173 tablet_replica.cc:330] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3: stopping tablet replica
I20260620 14:48:36.417068 17173 raft_consensus.cc:2229] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:36.417182 17173 raft_consensus.cc:2258] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:36.417384 17140 ts_tablet_manager.cc:1905] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:36.417469 17174 ts_tablet_manager.cc:1905] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:36.420354 17173 ts_tablet_manager.cc:1905] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:36.420833 17140 ts_tablet_manager.cc:1918] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:48:36.420835 17174 ts_tablet_manager.cc:1918] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:48:36.420924 17140 log.cc:1196] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/56ee3fc6bd294119868f7a26d119ab43
I20260620 14:48:36.420924 17174 log.cc:1196] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/wal/wals/56ee3fc6bd294119868f7a26d119ab43
I20260620 14:48:36.421205 17174 ts_tablet_manager.cc:1939] T 56ee3fc6bd294119868f7a26d119ab43 P fa4709532c534b7ab9d6911dda04a269: Deleting consensus metadata
I20260620 14:48:36.421205 17140 ts_tablet_manager.cc:1939] T 56ee3fc6bd294119868f7a26d119ab43 P 4a1f5d566aa440298ff26dc1610cde12: Deleting consensus metadata
I20260620 14:48:36.422096 16589 catalog_manager.cc:4841] TS fa4709532c534b7ab9d6911dda04a269 (127.8.221.193:40715): tablet 56ee3fc6bd294119868f7a26d119ab43 (table test-table [id=d875e2f4716043658f7d52de80577b61]) successfully deleted
I20260620 14:48:36.422268 16591 catalog_manager.cc:4841] TS 4a1f5d566aa440298ff26dc1610cde12 (127.8.221.194:43599): tablet 56ee3fc6bd294119868f7a26d119ab43 (table test-table [id=d875e2f4716043658f7d52de80577b61]) successfully deleted
I20260620 14:48:36.424324 17173 ts_tablet_manager.cc:1918] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:48:36.424422 17173 log.cc:1196] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/56ee3fc6bd294119868f7a26d119ab43
I20260620 14:48:36.424680 17173 ts_tablet_manager.cc:1939] T 56ee3fc6bd294119868f7a26d119ab43 P c82552077c6e41a5b1c4bcb154b1ffa3: Deleting consensus metadata
I20260620 14:48:36.425339 16591 catalog_manager.cc:4841] TS c82552077c6e41a5b1c4bcb154b1ffa3 (127.8.221.195:38933): tablet 56ee3fc6bd294119868f7a26d119ab43 (table test-table [id=d875e2f4716043658f7d52de80577b61]) successfully deleted
Jun 20 14:48:36 dist-test-slave-65tz krb5kdc[16560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:48:36 dist-test-slave-65tz krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:36 dist-test-slave-65tz krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:48:36.447726 16654 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:36.441524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.8.221.254:33969)
0620 14:48:36.441657 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:36.441662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:36.441966 (+ 304us) server_negotiation.cc:407] Connection header received
0620 14:48:36.442116 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:36.442118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:36.442159 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:36.442247 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:36.442931 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.443452 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.444299 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.444515 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.444625 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:36.447593 (+ 2968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54424: BlockingRecv error: recv got EOF from 127.0.0.1:54424 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 14:48:36 dist-test-slave-65tz krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM
I20260620 14:48:36.458237 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:36.449827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.8.221.254:33969)
0620 14:48:36.449923 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:36.449927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:36.450186 (+ 259us) server_negotiation.cc:407] Connection header received
0620 14:48:36.450351 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:36.450353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:36.450394 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:36.450488 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:36.451042 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.451529 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.452284 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.452509 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.452614 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:36.454880 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:36.454884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:36.454887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:36.454910 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:36.456569 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:36.456977 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:36.456982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:36.456984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:36.457066 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:36.457342 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:36.457345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:36.457347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:36.457526 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:36.457656 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:36.457940 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 14:48:36.458047 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:48:36.460938 16600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:48:36.461409 16600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:36.466229 16742 tablet_service.cc:1467] Processing CreateTablet for tablet e2dba50bb8984b79a218f649384ea443 (DEFAULT_TABLE table=test-table [id=1fc268bba0084b758bee7b3d357a17df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:36.466557 16742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2dba50bb8984b79a218f649384ea443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:36.466495 16869 tablet_service.cc:1467] Processing CreateTablet for tablet e2dba50bb8984b79a218f649384ea443 (DEFAULT_TABLE table=test-table [id=1fc268bba0084b758bee7b3d357a17df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:36.466745 16869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2dba50bb8984b79a218f649384ea443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:36.469344 17118 tablet_bootstrap.cc:492] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12: Bootstrap starting.
I20260620 14:48:36.469489 17116 tablet_bootstrap.cc:492] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269: Bootstrap starting.
I20260620 14:48:36.470340 17118 tablet_bootstrap.cc:654] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:36.470404 17116 tablet_bootstrap.cc:654] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:36.471232 17022 tablet_service.cc:1467] Processing CreateTablet for tablet e2dba50bb8984b79a218f649384ea443 (DEFAULT_TABLE table=test-table [id=1fc268bba0084b758bee7b3d357a17df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:36.471513 17022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2dba50bb8984b79a218f649384ea443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:36.472162 17116 tablet_bootstrap.cc:492] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269: No bootstrap required, opened a new log
I20260620 14:48:36.472240 17116 ts_tablet_manager.cc:1397] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:36.472698 17116 raft_consensus.cc:348] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.472818 17116 raft_consensus.cc:374] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:36.472849 17116 raft_consensus.cc:729] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4709532c534b7ab9d6911dda04a269, State: Initialized, Role: FOLLOWER
I20260620 14:48:36.472975 17116 consensus_queue.cc:260] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.473248 17116 ts_tablet_manager.cc:1428] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:36.474148 17117 tablet_bootstrap.cc:492] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3: Bootstrap starting.
I20260620 14:48:36.475337 17117 tablet_bootstrap.cc:654] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:36.476305 17118 tablet_bootstrap.cc:492] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12: No bootstrap required, opened a new log
I20260620 14:48:36.476394 17118 ts_tablet_manager.cc:1397] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 14:48:36.476848 17118 raft_consensus.cc:348] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.476975 17118 raft_consensus.cc:374] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:36.477016 17118 raft_consensus.cc:729] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1f5d566aa440298ff26dc1610cde12, State: Initialized, Role: FOLLOWER
I20260620 14:48:36.477170 17118 consensus_queue.cc:260] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.477438 17118 ts_tablet_manager.cc:1428] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:36.477466 17117 tablet_bootstrap.cc:492] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3: No bootstrap required, opened a new log
I20260620 14:48:36.477542 17117 ts_tablet_manager.cc:1397] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:36.478073 17117 raft_consensus.cc:348] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.478194 17117 raft_consensus.cc:374] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:36.478229 17117 raft_consensus.cc:729] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82552077c6e41a5b1c4bcb154b1ffa3, State: Initialized, Role: FOLLOWER
I20260620 14:48:36.478574 17117 consensus_queue.cc:260] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.478837 17117 ts_tablet_manager.cc:1428] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:36.690707 17123 raft_consensus.cc:482] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:36.690920 17123 raft_consensus.cc:504] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.691498 17123 leader_election.cc:288] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa4709532c534b7ab9d6911dda04a269 (127.8.221.193:40715), c82552077c6e41a5b1c4bcb154b1ffa3 (127.8.221.195:38933)
I20260620 14:48:36.692180 16762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2dba50bb8984b79a218f649384ea443" candidate_uuid: "4a1f5d566aa440298ff26dc1610cde12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4709532c534b7ab9d6911dda04a269" is_pre_election: true
I20260620 14:48:36.692358 16762 raft_consensus.cc:2454] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1f5d566aa440298ff26dc1610cde12 in term 0.
I20260620 14:48:36.692374 17042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2dba50bb8984b79a218f649384ea443" candidate_uuid: "4a1f5d566aa440298ff26dc1610cde12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" is_pre_election: true
I20260620 14:48:36.692682 17042 raft_consensus.cc:2454] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1f5d566aa440298ff26dc1610cde12 in term 0.
I20260620 14:48:36.692759 16833 leader_election.cc:302] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1f5d566aa440298ff26dc1610cde12, fa4709532c534b7ab9d6911dda04a269; no voters:
I20260620 14:48:36.692974 17123 raft_consensus.cc:2781] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:36.693029 17123 raft_consensus.cc:482] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:36.693058 17123 raft_consensus.cc:3037] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:36.694489 17123 raft_consensus.cc:504] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.694911 17123 leader_election.cc:288] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [CANDIDATE]: Term 1 election: Requested vote from peers fa4709532c534b7ab9d6911dda04a269 (127.8.221.193:40715), c82552077c6e41a5b1c4bcb154b1ffa3 (127.8.221.195:38933)
I20260620 14:48:36.695328 16762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2dba50bb8984b79a218f649384ea443" candidate_uuid: "4a1f5d566aa440298ff26dc1610cde12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa4709532c534b7ab9d6911dda04a269"
I20260620 14:48:36.695485 16762 raft_consensus.cc:3037] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:36.695497 17042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2dba50bb8984b79a218f649384ea443" candidate_uuid: "4a1f5d566aa440298ff26dc1610cde12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3"
I20260620 14:48:36.695672 17042 raft_consensus.cc:3037] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:36.696731 16762 raft_consensus.cc:2454] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1f5d566aa440298ff26dc1610cde12 in term 1.
I20260620 14:48:36.696743 17042 raft_consensus.cc:2454] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1f5d566aa440298ff26dc1610cde12 in term 1.
I20260620 14:48:36.697058 16833 leader_election.cc:302] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1f5d566aa440298ff26dc1610cde12, fa4709532c534b7ab9d6911dda04a269; no voters:
I20260620 14:48:36.697224 17123 raft_consensus.cc:2781] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:36.697345 17123 raft_consensus.cc:686] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 LEADER]: Becoming Leader. State: Replica: 4a1f5d566aa440298ff26dc1610cde12, State: Running, Role: LEADER
I20260620 14:48:36.697513 17123 consensus_queue.cc:237] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.699842 16603 catalog_manager.cc:5495] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1f5d566aa440298ff26dc1610cde12 (127.8.221.194). New cstate: current_term: 1 leader_uuid: "4a1f5d566aa440298ff26dc1610cde12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } health_report { overall_health: UNKNOWN } } }
I20260620 14:48:36.723667 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:36.720189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.8.221.194:43599)
0620 14:48:36.720406 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:36.720412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:36.720429 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:48:36.720515 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:36.720521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:36.720578 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:36.720667 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:36.720940 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.721516 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.722551 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.722778 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.722892 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:36.723167 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:36.723257 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:36.723342 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:36.723437 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:48:36.723490 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260620 14:48:36.726081 16762 raft_consensus.cc:1261] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269 [term 1 FOLLOWER]: Refusing update from remote peer 4a1f5d566aa440298ff26dc1610cde12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:36.726094 17042 raft_consensus.cc:1261] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 1 FOLLOWER]: Refusing update from remote peer 4a1f5d566aa440298ff26dc1610cde12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:36.726555 17126 consensus_queue.cc:1035] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:36.726763 17123 consensus_queue.cc:1035] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:36.735589 16605 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:36.735765 16605 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:36.737970 16605 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885: Sending DeleteTablet for 3 replicas of tablet e2dba50bb8984b79a218f649384ea443
I20260620 14:48:36.738682 16742 tablet_service.cc:1514] Processing DeleteTablet for tablet e2dba50bb8984b79a218f649384ea443 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:36 UTC) from {username='oryx', principal='oryx/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:51172
I20260620 14:48:36.738677 16869 tablet_service.cc:1514] Processing DeleteTablet for tablet e2dba50bb8984b79a218f649384ea443 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:36 UTC) from {username='oryx', principal='oryx/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:54372
I20260620 14:48:36.738772 17022 tablet_service.cc:1514] Processing DeleteTablet for tablet e2dba50bb8984b79a218f649384ea443 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:36 UTC) from {username='oryx', principal='oryx/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:59296
I20260620 14:48:36.738930 17174 tablet_replica.cc:330] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269: stopping tablet replica
I20260620 14:48:36.739010 17140 tablet_replica.cc:330] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12: stopping tablet replica
I20260620 14:48:36.739084 17174 raft_consensus.cc:2229] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:36.739126 17140 raft_consensus.cc:2229] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:36.739254 17174 raft_consensus.cc:2258] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:36.739368 17140 raft_consensus.cc:2258] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:36.739564 17173 tablet_replica.cc:330] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3: stopping tablet replica
I20260620 14:48:36.739684 17173 raft_consensus.cc:2229] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:36.739838 17173 raft_consensus.cc:2258] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:36.740499 17174 ts_tablet_manager.cc:1905] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:36.740545 17140 ts_tablet_manager.cc:1905] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:36.740545 17173 ts_tablet_manager.cc:1905] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:36.742911 17174 ts_tablet_manager.cc:1918] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:36.743018 17174 log.cc:1196] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-0/wal/wals/e2dba50bb8984b79a218f649384ea443
I20260620 14:48:36.743016 17140 ts_tablet_manager.cc:1918] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:36.743016 17173 ts_tablet_manager.cc:1918] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:36.743091 17140 log.cc:1196] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/e2dba50bb8984b79a218f649384ea443
I20260620 14:48:36.743091 17173 log.cc:1196] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/e2dba50bb8984b79a218f649384ea443
I20260620 14:48:36.743324 17174 ts_tablet_manager.cc:1939] T e2dba50bb8984b79a218f649384ea443 P fa4709532c534b7ab9d6911dda04a269: Deleting consensus metadata
I20260620 14:48:36.743402 17173 ts_tablet_manager.cc:1939] T e2dba50bb8984b79a218f649384ea443 P c82552077c6e41a5b1c4bcb154b1ffa3: Deleting consensus metadata
I20260620 14:48:36.743417 17140 ts_tablet_manager.cc:1939] T e2dba50bb8984b79a218f649384ea443 P 4a1f5d566aa440298ff26dc1610cde12: Deleting consensus metadata
I20260620 14:48:36.743275 16605 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:48:36.744057 16591 catalog_manager.cc:4841] TS 4a1f5d566aa440298ff26dc1610cde12 (127.8.221.194:43599): tablet e2dba50bb8984b79a218f649384ea443 (table test-table [id=1fc268bba0084b758bee7b3d357a17df]) successfully deleted
I20260620 14:48:36.744263 16591 catalog_manager.cc:4841] TS c82552077c6e41a5b1c4bcb154b1ffa3 (127.8.221.195:38933): tablet e2dba50bb8984b79a218f649384ea443 (table test-table [id=1fc268bba0084b758bee7b3d357a17df]) successfully deleted
I20260620 14:48:36.744293 16589 catalog_manager.cc:4841] TS fa4709532c534b7ab9d6911dda04a269 (127.8.221.193:40715): tablet e2dba50bb8984b79a218f649384ea443 (table test-table [id=1fc268bba0084b758bee7b3d357a17df]) successfully deleted
W20260620 14:48:36.744448 16605 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:36.749917 16742 tablet_service.cc:1467] Processing CreateTablet for tablet 95a543e6c7a143bea2fefe534b1765d4 (DEFAULT_TABLE table=test-table [id=9e1640f31eb74556a88ea97ab048ba7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:36.750005 17022 tablet_service.cc:1467] Processing CreateTablet for tablet 95a543e6c7a143bea2fefe534b1765d4 (DEFAULT_TABLE table=test-table [id=9e1640f31eb74556a88ea97ab048ba7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:36.750265 16742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a543e6c7a143bea2fefe534b1765d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:36.750368 17022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a543e6c7a143bea2fefe534b1765d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:36.752338 17116 tablet_bootstrap.cc:492] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269: Bootstrap starting.
I20260620 14:48:36.752578 16869 tablet_service.cc:1467] Processing CreateTablet for tablet 95a543e6c7a143bea2fefe534b1765d4 (DEFAULT_TABLE table=test-table [id=9e1640f31eb74556a88ea97ab048ba7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:36.752818 16869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a543e6c7a143bea2fefe534b1765d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:36.753219 17116 tablet_bootstrap.cc:654] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:36.754459 17116 tablet_bootstrap.cc:492] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269: No bootstrap required, opened a new log
I20260620 14:48:36.754554 17116 ts_tablet_manager.cc:1397] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:48:36.754971 17116 raft_consensus.cc:348] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.755101 17116 raft_consensus.cc:374] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:36.755154 17116 raft_consensus.cc:729] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa4709532c534b7ab9d6911dda04a269, State: Initialized, Role: FOLLOWER
I20260620 14:48:36.755293 17117 tablet_bootstrap.cc:492] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3: Bootstrap starting.
I20260620 14:48:36.755272 17116 consensus_queue.cc:260] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.755538 17116 ts_tablet_manager.cc:1428] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:36.756244 17117 tablet_bootstrap.cc:654] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:36.756894 17118 tablet_bootstrap.cc:492] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12: Bootstrap starting.
I20260620 14:48:36.757730 17118 tablet_bootstrap.cc:654] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:36.757725 17117 tablet_bootstrap.cc:492] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3: No bootstrap required, opened a new log
I20260620 14:48:36.757920 17117 ts_tablet_manager.cc:1397] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:36.758553 17117 raft_consensus.cc:348] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.758715 17117 raft_consensus.cc:374] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:36.758764 17117 raft_consensus.cc:729] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82552077c6e41a5b1c4bcb154b1ffa3, State: Initialized, Role: FOLLOWER
I20260620 14:48:36.758961 17117 consensus_queue.cc:260] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.759056 17118 tablet_bootstrap.cc:492] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12: No bootstrap required, opened a new log
I20260620 14:48:36.759136 17118 ts_tablet_manager.cc:1397] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:48:36.759371 17117 ts_tablet_manager.cc:1428] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:36.759528 17118 raft_consensus.cc:348] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.759658 17118 raft_consensus.cc:374] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:36.759702 17118 raft_consensus.cc:729] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1f5d566aa440298ff26dc1610cde12, State: Initialized, Role: FOLLOWER
I20260620 14:48:36.759826 17118 consensus_queue.cc:260] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:36.760121 17118 ts_tablet_manager.cc:1428] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:48:37.116520 17146 raft_consensus.cc:482] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:37.116703 17146 raft_consensus.cc:504] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:37.117281 17146 leader_election.cc:288] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1f5d566aa440298ff26dc1610cde12 (127.8.221.194:43599), c82552077c6e41a5b1c4bcb154b1ffa3 (127.8.221.195:38933)
I20260620 14:48:37.117861 16902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a543e6c7a143bea2fefe534b1765d4" candidate_uuid: "fa4709532c534b7ab9d6911dda04a269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1f5d566aa440298ff26dc1610cde12" is_pre_election: true
I20260620 14:48:37.117879 17042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a543e6c7a143bea2fefe534b1765d4" candidate_uuid: "fa4709532c534b7ab9d6911dda04a269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" is_pre_election: true
I20260620 14:48:37.118046 16902 raft_consensus.cc:2454] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa4709532c534b7ab9d6911dda04a269 in term 0.
I20260620 14:48:37.118093 17042 raft_consensus.cc:2454] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa4709532c534b7ab9d6911dda04a269 in term 0.
I20260620 14:48:37.118432 16695 leader_election.cc:302] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c82552077c6e41a5b1c4bcb154b1ffa3, fa4709532c534b7ab9d6911dda04a269; no voters:
I20260620 14:48:37.118639 17146 raft_consensus.cc:2781] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:37.118718 17146 raft_consensus.cc:482] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:37.118774 17146 raft_consensus.cc:3037] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:37.119884 17146 raft_consensus.cc:504] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:37.120280 17146 leader_election.cc:288] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [CANDIDATE]: Term 1 election: Requested vote from peers 4a1f5d566aa440298ff26dc1610cde12 (127.8.221.194:43599), c82552077c6e41a5b1c4bcb154b1ffa3 (127.8.221.195:38933)
I20260620 14:48:37.120734 16902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a543e6c7a143bea2fefe534b1765d4" candidate_uuid: "fa4709532c534b7ab9d6911dda04a269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1f5d566aa440298ff26dc1610cde12"
I20260620 14:48:37.120766 17042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a543e6c7a143bea2fefe534b1765d4" candidate_uuid: "fa4709532c534b7ab9d6911dda04a269" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3"
I20260620 14:48:37.120869 16902 raft_consensus.cc:3037] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:37.120859 17042 raft_consensus.cc:3037] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:37.121902 17042 raft_consensus.cc:2454] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa4709532c534b7ab9d6911dda04a269 in term 1.
I20260620 14:48:37.122157 16902 raft_consensus.cc:2454] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa4709532c534b7ab9d6911dda04a269 in term 1.
I20260620 14:48:37.122231 16695 leader_election.cc:302] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c82552077c6e41a5b1c4bcb154b1ffa3, fa4709532c534b7ab9d6911dda04a269; no voters:
I20260620 14:48:37.122396 17146 raft_consensus.cc:2781] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:37.122596 17146 raft_consensus.cc:686] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [term 1 LEADER]: Becoming Leader. State: Replica: fa4709532c534b7ab9d6911dda04a269, State: Running, Role: LEADER
I20260620 14:48:37.122753 17146 consensus_queue.cc:237] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } }
I20260620 14:48:37.124419 16603 catalog_manager.cc:5495] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa4709532c534b7ab9d6911dda04a269 (127.8.221.193). New cstate: current_term: 1 leader_uuid: "fa4709532c534b7ab9d6911dda04a269" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa4709532c534b7ab9d6911dda04a269" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 } health_report { overall_health: UNKNOWN } } }
I20260620 14:48:37.219913 17201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:37.216605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51190 (local address 127.8.221.193:40715)
0620 14:48:37.216886 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:37.216890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:37.216905 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:48:37.216952 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:37.216954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:37.216998 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:37.217094 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:37.217411 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.217966 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.218919 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.219136 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.219222 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:37.219493 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:37.219564 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:37.219623 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:37.219739 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 14:48:37.219779 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":117,"threads_started":1}
I20260620 14:48:37.226904 17042 raft_consensus.cc:1261] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 1 FOLLOWER]: Refusing update from remote peer fa4709532c534b7ab9d6911dda04a269: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:37.226904 16902 raft_consensus.cc:1261] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 FOLLOWER]: Refusing update from remote peer fa4709532c534b7ab9d6911dda04a269: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:37.227350 17146 consensus_queue.cc:1035] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1f5d566aa440298ff26dc1610cde12" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:37.227535 17199 consensus_queue.cc:1035] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82552077c6e41a5b1c4bcb154b1ffa3" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:37.261536 16603 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:37.261690 16603 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:37.263842 16603 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d601664b05e64c2085b8323d3f39a885: Sending DeleteTablet for 3 replicas of tablet 95a543e6c7a143bea2fefe534b1765d4
I20260620 14:48:37.264557 16742 tablet_service.cc:1514] Processing DeleteTablet for tablet 95a543e6c7a143bea2fefe534b1765d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:37 UTC) from {username='oryx', principal='oryx/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:51172
I20260620 14:48:37.264634 16869 tablet_service.cc:1514] Processing DeleteTablet for tablet 95a543e6c7a143bea2fefe534b1765d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:37 UTC) from {username='oryx', principal='oryx/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:54372
I20260620 14:48:37.264727 17022 tablet_service.cc:1514] Processing DeleteTablet for tablet 95a543e6c7a143bea2fefe534b1765d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:37 UTC) from {username='oryx', principal='oryx/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:59296
I20260620 14:48:37.265000 17208 tablet_replica.cc:330] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12: stopping tablet replica
I20260620 14:48:37.265093 17209 tablet_replica.cc:330] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3: stopping tablet replica
I20260620 14:48:37.265146 17207 tablet_replica.cc:330] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269: stopping tablet replica
I20260620 14:48:37.265156 17208 raft_consensus.cc:2229] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:37.265276 17208 raft_consensus.cc:2258] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:37.265268 17207 raft_consensus.cc:2229] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:37.265303 17209 raft_consensus.cc:2229] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:37.265470 17207 raft_consensus.cc:2258] T 95a543e6c7a143bea2fefe534b1765d4 P fa4709532c534b7ab9d6911dda04a269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:37.265492 17209 raft_consensus.cc:2258] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:37.266094 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 16677
I20260620 14:48:37.266244 17208 ts_tablet_manager.cc:1905] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:37.266770 17209 ts_tablet_manager.cc:1905] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:37.268747 17208 ts_tablet_manager.cc:1918] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:48:37.268852 17208 log.cc:1196] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/95a543e6c7a143bea2fefe534b1765d4
I20260620 14:48:37.269133 17208 ts_tablet_manager.cc:1939] T 95a543e6c7a143bea2fefe534b1765d4 P 4a1f5d566aa440298ff26dc1610cde12: Deleting consensus metadata
I20260620 14:48:37.269775 16591 catalog_manager.cc:4841] TS 4a1f5d566aa440298ff26dc1610cde12 (127.8.221.194:43599): tablet 95a543e6c7a143bea2fefe534b1765d4 (table test-table [id=9e1640f31eb74556a88ea97ab048ba7c]) successfully deleted
I20260620 14:48:37.270476 17209 ts_tablet_manager.cc:1918] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:48:37.270576 17209 log.cc:1196] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/95a543e6c7a143bea2fefe534b1765d4
I20260620 14:48:37.270848 17209 ts_tablet_manager.cc:1939] T 95a543e6c7a143bea2fefe534b1765d4 P c82552077c6e41a5b1c4bcb154b1ffa3: Deleting consensus metadata
I20260620 14:48:37.271587 16591 catalog_manager.cc:4841] TS c82552077c6e41a5b1c4bcb154b1ffa3 (127.8.221.195:38933): tablet 95a543e6c7a143bea2fefe534b1765d4 (table test-table [id=9e1640f31eb74556a88ea97ab048ba7c]) successfully deleted
W20260620 14:48:37.276413 16589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.221.193:40715 (error 108)
I20260620 14:48:37.276867 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 16817
I20260620 14:48:37.278679 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:37.278223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:40715 (local address 127.0.0.1:51194)
0620 14:48:37.278466 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:37.278552 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.193:40715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":77,"threads_started":1}
W20260620 14:48:37.279181 16589 catalog_manager.cc:4584] TS fa4709532c534b7ab9d6911dda04a269 (127.8.221.193:40715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 95a543e6c7a143bea2fefe534b1765d4: Network error: Client connection negotiation failed: client connection to 127.8.221.193:40715: connect: Connection refused (error 111)
I20260620 14:48:37.286242 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 16957
I20260620 14:48:37.293013 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 16573
2026-06-20T14:48:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3946 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:37 dist-test-slave-65tz krb5kdc[17214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:37 dist-test-slave-65tz krb5kdc[17214](info): set up 2 sockets
Jun 20 14:48:37 dist-test-slave-65tz krb5kdc[17214](info): commencing operation
krb5kdc: starting...
W20260620 14:48:38.439216 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:48:38 dist-test-slave-65tz krb5kdc[17214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:48:38.576977 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:33907
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:33907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:38.687633 17227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:38.687950 17227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:38.688019 17227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:38.692086 17227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:38.692171 17227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:38.692207 17227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:38.692261 17227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:38.692312 17227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:38.697633 17227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:33907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:33907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:38.698781 17227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:38.699800 17227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:38.708647 17235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:38.708647 17233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:38.708624 17232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:38.709215 17227 server_base.cc:1034] running on GCE node
I20260620 14:48:38.709945 17227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:38.711320 17227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:38.712553 17227 hybrid_clock.cc:648] HybridClock initialized: now 1781966918712527 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T14:48:38Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1428 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:38 dist-test-slave-65tz krb5kdc[17240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:38 dist-test-slave-65tz krb5kdc[17240](info): set up 2 sockets
Jun 20 14:48:38 dist-test-slave-65tz krb5kdc[17240](info): commencing operation
krb5kdc: starting...
W20260620 14:48:39.783591 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:48:39 dist-test-slave-65tz krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:48:39.919134 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:45155
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:38909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:45155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:40.057783 17253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:40.058288 17253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:40.058390 17253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:40.062960 17253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:40.063084 17253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:40.063120 17253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:40.063154 17253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:40.063189 17253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:40.069505 17253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:38909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:45155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:45155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:40.071089 17253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:40.072399 17253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:40.082813 17258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:40.082820 17261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:40.082860 17259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:40.083308 17253 server_base.cc:1034] running on GCE node
I20260620 14:48:40.083988 17253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:40.085326 17253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:40.086495 17253 hybrid_clock.cc:648] HybridClock initialized: now 1781966920086429 us; error 89 us; skew 500 ppm
Jun 20 14:48:40 dist-test-slave-65tz krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966920, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:40.089730 17253 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:48:40.090899 17253 webserver.cc:460] Webserver started at http://127.8.221.254:45971/ using document root <none> and password file <none>
I20260620 14:48:40.091591 17253 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:40.091657 17253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:40.091918 17253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:40.094537 17253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "56e658ec9cae4eaa8a12cabfd7af70f3"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-65tz"
server_key: "2373808059f512f69a6c0a8a96d8d513"
server_key_iv: "b555dee1069208fc14893ce04cdba8e8"
server_key_version: "encryptionkey@0"
I20260620 14:48:40.095202 17253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "56e658ec9cae4eaa8a12cabfd7af70f3"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-65tz"
server_key: "2373808059f512f69a6c0a8a96d8d513"
server_key_iv: "b555dee1069208fc14893ce04cdba8e8"
server_key_version: "encryptionkey@0"
I20260620 14:48:40.100045 17253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:48:40.103083 17268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:40.104904 17253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:40.105104 17253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "56e658ec9cae4eaa8a12cabfd7af70f3"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-65tz"
server_key: "2373808059f512f69a6c0a8a96d8d513"
server_key_iv: "b555dee1069208fc14893ce04cdba8e8"
server_key_version: "encryptionkey@0"
I20260620 14:48:40.105255 17253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:40.131035 17253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:40.134733 17253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:40.134981 17253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:40.144208 17253 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:45155
I20260620 14:48:40.144280 17320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:45155 every 8 connection(s)
I20260620 14:48:40.145555 17253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:40.147637 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 17253
I20260620 14:48:40.147797 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:48:40.148124 9079 external_mini_cluster.cc:1428] Setting key 0959aaaa73df38dcb04620a0bcf2ff39
I20260620 14:48:40.149261 17321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:40.156677 17321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3: Bootstrap starting.
Jun 20 14:48:40 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:40.159500 17321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:40.160233 17321 log.cc:824] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:40.162597 17321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3: No bootstrap required, opened a new log
I20260620 14:48:40.165587 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.149954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42406 (local address 127.8.221.254:45155)
0620 14:48:40.150515 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:40.150539 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:40.150596 (+ 57us) server_negotiation.cc:407] Connection header received
0620 14:48:40.151428 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:40.151451 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:40.151866 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:40.152253 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:40.153245 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.154470 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.155769 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.156374 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.156536 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.159754 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:40.159760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:40.159777 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:40.159811 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:40.162024 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:40.162549 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:40.162554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:40.162560 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:40.162683 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:40.163169 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:40.163172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:40.163173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:40.163662 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:40.163847 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:40.164193 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 14:48:40.164621 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":207,"threads_started":1}
I20260620 14:48:40.166946 17321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e658ec9cae4eaa8a12cabfd7af70f3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 45155 } }
I20260620 14:48:40.167218 17321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:40.167300 17321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e658ec9cae4eaa8a12cabfd7af70f3, State: Initialized, Role: FOLLOWER
I20260620 14:48:40.167809 17321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e658ec9cae4eaa8a12cabfd7af70f3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 45155 } }
I20260620 14:48:40.167968 17321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:40.168059 17321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:40.168167 17321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:40.169150 17321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e658ec9cae4eaa8a12cabfd7af70f3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 45155 } }
I20260620 14:48:40.169529 17321 leader_election.cc:302] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56e658ec9cae4eaa8a12cabfd7af70f3; no voters:
I20260620 14:48:40.169929 17321 leader_election.cc:288] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:40.170137 17326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:40.170548 17326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [term 1 LEADER]: Becoming Leader. State: Replica: 56e658ec9cae4eaa8a12cabfd7af70f3, State: Running, Role: LEADER
I20260620 14:48:40.170948 17326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e658ec9cae4eaa8a12cabfd7af70f3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 45155 } }
I20260620 14:48:40.171296 17321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:40.173184 17327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56e658ec9cae4eaa8a12cabfd7af70f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e658ec9cae4eaa8a12cabfd7af70f3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 45155 } } }
I20260620 14:48:40.173347 17327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:40.173609 17328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56e658ec9cae4eaa8a12cabfd7af70f3. Latest consensus state: current_term: 1 leader_uuid: "56e658ec9cae4eaa8a12cabfd7af70f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e658ec9cae4eaa8a12cabfd7af70f3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 45155 } } }
I20260620 14:48:40.174181 17340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:40.174207 17328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:40.177393 17340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:40.184108 17340 catalog_manager.cc:1347] Generated new cluster ID: 4f3ba8ba411c4fcd836b9fcf55c3493d
I20260620 14:48:40.184221 17340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:40.194540 17340 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:40.195990 17340 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:40.205453 17340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3: Generated new TSK 0
I20260620 14:48:40.206399 17340 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
I20260620 14:48:40.245246 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:45155
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:38909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:40.377295 17347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:40.377632 17347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:40.377682 17347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:40.382045 17347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:40.382133 17347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:40.382257 17347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:48:40.388386 17347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:38909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:40.389850 17347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:40.391168 17347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:40.402570 17355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:40.402570 17353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:40.402593 17352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:40.403055 17347 server_base.cc:1034] running on GCE node
I20260620 14:48:40.403601 17347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:40.404393 17347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:40.405607 17347 hybrid_clock.cc:648] HybridClock initialized: now 1781966920405580 us; error 47 us; skew 500 ppm
Jun 20 14:48:40 dist-test-slave-65tz krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966920, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:40.409053 17347 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:48:40.410468 17347 webserver.cc:460] Webserver started at http://127.8.221.193:41355/ using document root <none> and password file <none>
I20260620 14:48:40.411269 17347 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:40.411346 17347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:40.411615 17347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:40.414321 17347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "311d452bea6c4e5bbb2bc227f5c4e7be"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-65tz"
server_key: "55ad048094eb5b1f1f6ef792c4121f70"
server_key_iv: "81e31289d6f2bf200b0b44ef60491431"
server_key_version: "encryptionkey@0"
I20260620 14:48:40.415030 17347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "311d452bea6c4e5bbb2bc227f5c4e7be"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-65tz"
server_key: "55ad048094eb5b1f1f6ef792c4121f70"
server_key_iv: "81e31289d6f2bf200b0b44ef60491431"
server_key_version: "encryptionkey@0"
I20260620 14:48:40.419973 17347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:48:40.423308 17362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:40.424917 17347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:48:40.425091 17347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "311d452bea6c4e5bbb2bc227f5c4e7be"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-65tz"
server_key: "55ad048094eb5b1f1f6ef792c4121f70"
server_key_iv: "81e31289d6f2bf200b0b44ef60491431"
server_key_version: "encryptionkey@0"
I20260620 14:48:40.425225 17347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:40.452777 17347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:40.456929 17347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:40.457201 17347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:40.458060 17347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:40.459141 17347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:40.459223 17347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:40.459311 17347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:40.459364 17347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:40.472055 17347 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:37785
I20260620 14:48:40.472118 17475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:37785 every 8 connection(s)
I20260620 14:48:40.473322 17347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:40.473912 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 17347
I20260620 14:48:40.474071 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:48:40.474517 9079 external_mini_cluster.cc:1428] Setting key 7f872eaabec171353544ddb8ee38355a
Jun 20 14:48:40 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966920, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.194@KRBTEST.COM" created.
I20260620 14:48:40.490036 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.475945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:39953 (local address 127.8.221.254:45155)
0620 14:48:40.476090 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:40.476093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:40.477948 (+ 1855us) server_negotiation.cc:407] Connection header received
0620 14:48:40.479159 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:40.479162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:40.479222 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:40.479307 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:40.480582 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.481143 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.482651 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.482908 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.483026 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.485833 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:40.485840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:40.485845 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:40.485878 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:40.487461 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:40.488373 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:40.488376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:40.488378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:40.488465 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:40.488869 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:40.488872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:40.488874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:40.489069 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:40.489180 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:40.489714 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 14:48:40.489868 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:48:40.491276 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.476708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.193:39953)
0620 14:48:40.477700 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:40.477769 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:40.478927 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:40.479492 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:40.479500 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:40.479879 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:40.480416 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:40.480429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.481273 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.481278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:40.482392 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:40.482401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.483075 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.483080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:40.483268 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.484015 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:40.484032 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:40.485587 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:40.487690 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:40.487698 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:40.487721 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:40.488177 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:40.488579 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:40.488585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:40.488592 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:40.488754 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:40.489178 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:40.489186 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:40.489457 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 14:48:40.489653 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 14:48:40.490031 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":153,"threads_started":1}
I20260620 14:48:40.492818 17476 heartbeater.cc:347] Connected to a master server at 127.8.221.254:45155
I20260620 14:48:40.493213 17476 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:40.494203 17476 heartbeater.cc:511] Master 127.8.221.254:45155 requested a full tablet report, sending...
I20260620 14:48:40.497040 17281 ts_manager.cc:194] Registered new tserver with Master: 311d452bea6c4e5bbb2bc227f5c4e7be (127.8.221.193:37785)
I20260620 14:48:40.498370 17281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:39953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
I20260620 14:48:40.509404 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:45155
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:38909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:40.638429 17481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:40.638777 17481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:40.638863 17481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:40.643049 17481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:40.643151 17481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:40.643249 17481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:48:40.649093 17481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:38909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:40.650561 17481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:40.651841 17481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:40.663111 17489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:40.663059 17487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:40.663045 17486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:40.663393 17481 server_base.cc:1034] running on GCE node
I20260620 14:48:40.664005 17481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:40.664888 17481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:40.666105 17481 hybrid_clock.cc:648] HybridClock initialized: now 1781966920666055 us; error 64 us; skew 500 ppm
Jun 20 14:48:40 dist-test-slave-65tz krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966920, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:40.669745 17481 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:48:40.670917 17481 webserver.cc:460] Webserver started at http://127.8.221.194:45047/ using document root <none> and password file <none>
I20260620 14:48:40.671634 17481 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:40.671701 17481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:40.671931 17481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:40.674525 17481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "df8b98e31b484377a4d831b16d4d1918"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-65tz"
server_key: "53bce856ee7c85bbf9406f6465c9a844"
server_key_iv: "d91d71ba7f268f49206630d9b2331f70"
server_key_version: "encryptionkey@0"
I20260620 14:48:40.675053 17481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "df8b98e31b484377a4d831b16d4d1918"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-65tz"
server_key: "53bce856ee7c85bbf9406f6465c9a844"
server_key_iv: "d91d71ba7f268f49206630d9b2331f70"
server_key_version: "encryptionkey@0"
I20260620 14:48:40.679345 17481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:48:40.682281 17496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:40.683773 17481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:40.683892 17481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "df8b98e31b484377a4d831b16d4d1918"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-65tz"
server_key: "53bce856ee7c85bbf9406f6465c9a844"
server_key_iv: "d91d71ba7f268f49206630d9b2331f70"
server_key_version: "encryptionkey@0"
I20260620 14:48:40.684053 17481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:40.703516 17481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:40.706943 17481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:40.707187 17481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:40.707803 17481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:40.708664 17481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:40.708736 17481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:40.708813 17481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:40.708858 17481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:40.718971 17481 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:45799
I20260620 14:48:40.718997 17609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:45799 every 8 connection(s)
I20260620 14:48:40.719934 17481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:40.727630 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 17481
I20260620 14:48:40.727767 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:40.728067 9079 external_mini_cluster.cc:1428] Setting key 7996c27cc456af91d36a454e4fe3826e
Jun 20 14:48:40 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966920, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:40.737627 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.722237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:48541 (local address 127.8.221.254:45155)
0620 14:48:40.722422 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:40.722428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:40.723601 (+ 1173us) server_negotiation.cc:407] Connection header received
0620 14:48:40.724725 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:40.724728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:40.724785 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:40.724876 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:40.726494 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.727013 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.728611 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.728838 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.728956 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.732617 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:40.732622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:40.732628 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:40.732668 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:40.734755 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:40.735471 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:40.735478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:40.735483 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:40.735583 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:40.736067 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:40.736071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:40.736073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:40.736363 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:40.736495 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:40.737227 (+ 732us) server_negotiation.cc:299] Negotiation successful
0620 14:48:40.737384 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:48:40.738767 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.722592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.194:48541)
0620 14:48:40.723401 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:40.723457 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:40.724476 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:40.725129 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:40.725136 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:40.725608 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:40.726340 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:40.726358 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.727143 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.727146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:40.728476 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:40.728487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.729077 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.729087 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:40.729288 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.729932 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:40.729949 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:40.732296 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:40.734926 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:40.734934 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:40.734953 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:40.735314 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:40.735730 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:40.735733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:40.735736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:40.735934 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:40.736503 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:40.736509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:40.736904 (+ 395us) client_negotiation.cc:769] Sending connection context
0620 14:48:40.737131 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 14:48:40.737534 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":128,"threads_started":1}
WARNING: no policy specified for kudu/127.8.221.195@KRBTEST.COM; defaulting to no policy
I20260620 14:48:40.740408 17610 heartbeater.cc:347] Connected to a master server at 127.8.221.254:45155
I20260620 14:48:40.740700 17610 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:40.741487 17610 heartbeater.cc:511] Master 127.8.221.254:45155 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.195@KRBTEST.COM" created.
I20260620 14:48:40.743005 17281 ts_manager.cc:194] Registered new tserver with Master: df8b98e31b484377a4d831b16d4d1918 (127.8.221.194:45799)
I20260620 14:48:40.743633 17281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:48541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
I20260620 14:48:40.761759 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:45155
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:38909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:40.885059 17615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:40.885385 17615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:40.885468 17615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:40.889730 17615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:40.889882 17615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:40.890014 17615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:48:40.895448 17615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:38909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:40.896624 17615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:40.897692 17615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:40.908805 17621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:40.908900 17623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:40.908805 17620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:40.909129 17615 server_base.cc:1034] running on GCE node
I20260620 14:48:40.909881 17615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:40.910786 17615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:40.912019 17615 hybrid_clock.cc:648] HybridClock initialized: now 1781966920911986 us; error 56 us; skew 500 ppm
Jun 20 14:48:40 dist-test-slave-65tz krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966920, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:40.915490 17615 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:48:40.916615 17615 webserver.cc:460] Webserver started at http://127.8.221.195:35747/ using document root <none> and password file <none>
I20260620 14:48:40.917302 17615 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:40.917403 17615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:40.917665 17615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:40.920126 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "feb5d38eeb364cafa1a0921a602d07b4"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-65tz"
server_key: "20b9df7d4f9cae25483f534297e71e35"
server_key_iv: "5703c5d44e269e155b20524958f2848c"
server_key_version: "encryptionkey@0"
I20260620 14:48:40.920771 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "feb5d38eeb364cafa1a0921a602d07b4"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-65tz"
server_key: "20b9df7d4f9cae25483f534297e71e35"
server_key_iv: "5703c5d44e269e155b20524958f2848c"
server_key_version: "encryptionkey@0"
I20260620 14:48:40.925220 17615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:48:40.928366 17630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:40.930353 17615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:40.930529 17615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "feb5d38eeb364cafa1a0921a602d07b4"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-65tz"
server_key: "20b9df7d4f9cae25483f534297e71e35"
server_key_iv: "5703c5d44e269e155b20524958f2848c"
server_key_version: "encryptionkey@0"
I20260620 14:48:40.930683 17615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:40.948544 17615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:40.952183 17615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:40.952464 17615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:40.953223 17615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:40.954347 17615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:40.954422 17615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:40.954493 17615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:40.954535 17615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:40.965648 17615 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:38895
I20260620 14:48:40.965699 17743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:38895 every 8 connection(s)
I20260620 14:48:40.966846 17615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:40.969851 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 17615
I20260620 14:48:40.970014 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:40.970438 9079 external_mini_cluster.cc:1428] Setting key 0a93f55765b6840f62157968bdcd341f
Jun 20 14:48:40 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966920, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:48:40.982148 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.969246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:37087 (local address 127.8.221.254:45155)
0620 14:48:40.969397 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:40.969400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:40.971073 (+ 1673us) server_negotiation.cc:407] Connection header received
0620 14:48:40.972384 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:40.972388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:40.972446 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:40.972523 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:40.973965 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.974501 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.975745 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.975976 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.976088 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.978669 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:40.978675 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:40.978682 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:40.978704 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:40.980117 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:40.980626 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:40.980634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:40.980636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:40.980707 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:40.981044 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:40.981047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:40.981049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:40.981235 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:40.981349 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:40.981852 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 14:48:40.981975 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:48:40.982990 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.969902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.195:37087)
0620 14:48:40.970873 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:40.970939 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:40.972153 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:40.972756 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:40.972763 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:40.973258 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:40.973825 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:40.973839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.974628 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.974632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:40.975618 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:40.975627 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.976128 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.976132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:40.976315 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.976954 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:40.976971 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:40.978512 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:40.980241 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:40.980247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:40.980262 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:40.980512 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:40.980803 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:40.980807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:40.980809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:40.980940 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:40.981345 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:40.981351 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:40.981594 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 14:48:40.981782 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 14:48:40.982117 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":191,"threads_started":1}
I20260620 14:48:40.984486 17744 heartbeater.cc:347] Connected to a master server at 127.8.221.254:45155
I20260620 14:48:40.984745 17744 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:40.985313 17744 heartbeater.cc:511] Master 127.8.221.254:45155 requested a full tablet report, sending...
I20260620 14:48:40.986423 17281 ts_manager.cc:194] Registered new tserver with Master: feb5d38eeb364cafa1a0921a602d07b4 (127.8.221.195:38895)
I20260620 14:48:40.987172 17281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:37087
I20260620 14:48:40.997584 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.193.keytab.
Entry for principal oryx/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.193.keytab.
WARNING: no policy specified for oryx/127.8.221.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.221.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.194.keytab.
Entry for principal oryx/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.194.keytab.
WARNING: no policy specified for oryx/127.8.221.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.221.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.195.keytab.
Entry for principal oryx/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.195.keytab.
I20260620 14:48:41.085412 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 17347
I20260620 14:48:41.091845 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 17481
I20260620 14:48:41.097956 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 17615
I20260620 14:48:41.105129 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 17253
I20260620 14:48:41.113418 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:45155
--webserver_interface=127.8.221.254
--webserver_port=45971
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:38909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:45155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:41.226970 17753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:41.227252 17753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:41.227291 17753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:41.231261 17753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:41.231320 17753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:41.231339 17753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:41.231356 17753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:41.231372 17753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:41.237217 17753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:38909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:45155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:45155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=45971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:41.238272 17753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:41.239261 17753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:41.248227 17758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:41.248219 17761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:41.248219 17759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:41.248492 17753 server_base.cc:1034] running on GCE node
I20260620 14:48:41.249029 17753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:41.250208 17753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:41.251437 17753 hybrid_clock.cc:648] HybridClock initialized: now 1781966921251415 us; error 47 us; skew 500 ppm
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966921, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:41.254642 17753 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:48:41.255674 17753 webserver.cc:460] Webserver started at http://127.8.221.254:45971/ using document root <none> and password file <none>
I20260620 14:48:41.256328 17753 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:41.256445 17753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:41.260735 17753 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:41.263175 17768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:41.264732 17753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:48:41.264895 17753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "56e658ec9cae4eaa8a12cabfd7af70f3"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-65tz"
server_key: "2373808059f512f69a6c0a8a96d8d513"
server_key_iv: "b555dee1069208fc14893ce04cdba8e8"
server_key_version: "encryptionkey@0"
I20260620 14:48:41.265468 17753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:41.283192 17753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:41.286530 17753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:41.286747 17753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:41.294816 17753 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:45155
I20260620 14:48:41.295050 17820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:45155 every 8 connection(s)
I20260620 14:48:41.296303 17753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:41.301980 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 17753
I20260620 14:48:41.302060 17821 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:48:41.302448 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:37785
--local_ip_for_outbound_sockets=127.8.221.193
--tserver_master_addrs=127.8.221.254:45155
--webserver_port=41355
--webserver_interface=127.8.221.193
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:38909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.221.193
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:48:41.304359 17821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3: Bootstrap starting.
I20260620 14:48:41.325599 17821 log.cc:824] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:41.333698 17821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:48:41.334223 17821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3: Bootstrap complete.
I20260620 14:48:41.339556 17821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e658ec9cae4eaa8a12cabfd7af70f3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 45155 } }
I20260620 14:48:41.340085 17821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e658ec9cae4eaa8a12cabfd7af70f3, State: Initialized, Role: FOLLOWER
I20260620 14:48:41.340694 17821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e658ec9cae4eaa8a12cabfd7af70f3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 45155 } }
I20260620 14:48:41.340863 17821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:41.340967 17821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:41.341110 17821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:48:41.343693 17821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e658ec9cae4eaa8a12cabfd7af70f3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 45155 } }
I20260620 14:48:41.344103 17821 leader_election.cc:302] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56e658ec9cae4eaa8a12cabfd7af70f3; no voters:
I20260620 14:48:41.344568 17821 leader_election.cc:288] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:48:41.344681 17825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:48:41.344980 17825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [term 2 LEADER]: Becoming Leader. State: Replica: 56e658ec9cae4eaa8a12cabfd7af70f3, State: Running, Role: LEADER
I20260620 14:48:41.345438 17825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e658ec9cae4eaa8a12cabfd7af70f3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 45155 } }
I20260620 14:48:41.346175 17821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:41.347139 17826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "56e658ec9cae4eaa8a12cabfd7af70f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e658ec9cae4eaa8a12cabfd7af70f3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 45155 } } }
I20260620 14:48:41.347301 17826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:41.348168 17827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56e658ec9cae4eaa8a12cabfd7af70f3. Latest consensus state: current_term: 2 leader_uuid: "56e658ec9cae4eaa8a12cabfd7af70f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56e658ec9cae4eaa8a12cabfd7af70f3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 45155 } } }
I20260620 14:48:41.348280 17833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:41.348340 17827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:41.351804 17833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:41.352432 17833 catalog_manager.cc:1259] Loaded cluster ID: 4f3ba8ba411c4fcd836b9fcf55c3493d
I20260620 14:48:41.352495 17833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:41.354846 17833 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:41.355509 17833 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 56e658ec9cae4eaa8a12cabfd7af70f3: Loaded TSK: 0
I20260620 14:48:41.356845 17833 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:48:41.418066 17823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:41.418309 17823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:41.418354 17823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:41.422498 17823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:41.422567 17823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:41.422652 17823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:48:41.428069 17823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:38909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.193.keytab
--principal=oryx/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:37785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=41355
--tserver_master_addrs=127.8.221.254:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:41.429318 17823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:41.430410 17823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:41.441612 17850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:41.441602 17847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:41.441695 17823 server_base.cc:1034] running on GCE node
W20260620 14:48:41.442006 17848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:41.442667 17823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:41.443627 17823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:41.444844 17823 hybrid_clock.cc:648] HybridClock initialized: now 1781966921444802 us; error 56 us; skew 500 ppm
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966921, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:41.449010 17823 init.cc:377] Logged in from keytab as oryx/127.8.221.193@KRBTEST.COM (short username oryx)
I20260620 14:48:41.450423 17823 webserver.cc:460] Webserver started at http://127.8.221.193:41355/ using document root <none> and password file <none>
I20260620 14:48:41.451227 17823 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:41.451339 17823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:41.457240 17823 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:48:41.460412 17857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:41.462381 17823 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:41.462566 17823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "311d452bea6c4e5bbb2bc227f5c4e7be"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-65tz"
server_key: "55ad048094eb5b1f1f6ef792c4121f70"
server_key_iv: "81e31289d6f2bf200b0b44ef60491431"
server_key_version: "encryptionkey@0"
I20260620 14:48:41.463443 17823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:41.495906 17823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:41.501505 17823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:41.501751 17823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:41.502624 17823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:41.503931 17823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:41.503999 17823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:41.504060 17823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:41.504091 17823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:41.517251 17823 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:37785
I20260620 14:48:41.518025 17970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:37785 every 8 connection(s)
I20260620 14:48:41.518800 17823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:41.519352 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 17823
I20260620 14:48:41.519917 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:45799
--local_ip_for_outbound_sockets=127.8.221.194
--tserver_master_addrs=127.8.221.254:45155
--webserver_port=45047
--webserver_interface=127.8.221.194
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:38909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.221.194
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:48:41.533712 17975 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:41.521714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:47573 (local address 127.8.221.254:45155)
0620 14:48:41.522560 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:41.522588 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:41.523413 (+ 825us) server_negotiation.cc:407] Connection header received
0620 14:48:41.524609 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:41.524618 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:41.524925 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:41.525312 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:41.526573 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.527559 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.528726 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.529188 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.529371 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.532946 (+ 3575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.193:47573: BlockingRecv error: recv got EOF from 127.8.221.193:47573 (error 108)
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":154,"threads_started":1}
I20260620 14:48:41.533721 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:41.521853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.193:47573)
0620 14:48:41.523231 (+ 1378us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:41.523278 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:41.524178 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:41.525366 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:41.525374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:41.525915 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:41.526385 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.526399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.527702 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.527706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.528595 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.528605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.529363 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.529368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.529558 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.530227 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:41.530247 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:41.532735 (+ 2488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1111,"thread_start_us":171,"threads_started":1}
W20260620 14:48:41.534636 17971 heartbeater.cc:644] Failed to heartbeat to 127.8.221.254:45155 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.221.254:45155: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:41.542097 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:41.535280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.193:50869)
0620 14:48:41.535482 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:41.535501 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:41.535635 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:41.535953 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:41.535956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:41.536166 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:41.536328 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.536335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.537088 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.537091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.538222 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.538235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.538888 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.538891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.538992 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.539574 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:41.539589 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:41.541907 (+ 2318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
W20260620 14:48:41.542162 17975 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:41.535418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:50869 (local address 127.8.221.254:45155)
0620 14:48:41.535547 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:41.535550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:41.535574 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:48:41.535747 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:41.535749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:41.535811 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:41.535899 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:41.536453 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.536970 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.538379 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.538739 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.538873 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.542045 (+ 3172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.193:50869: BlockingRecv error: recv got EOF from 127.8.221.193:50869 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:41.548717 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:41.542809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.193:55329)
0620 14:48:41.542945 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:41.542963 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:41.543079 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:41.543381 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:41.543384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:41.543563 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:41.543654 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.543659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.544439 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.544446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.545036 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.545043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.545604 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.545608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.545704 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.546351 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:41.546363 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:41.548544 (+ 2181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 14:48:41.549036 17971 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:48:41.549249 17975 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:41.542960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:55329 (local address 127.8.221.254:45155)
0620 14:48:41.543171 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:41.543176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:41.543197 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:48:41.543229 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:41.543231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:41.543268 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:41.543360 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:41.543769 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.544322 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.545161 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.545467 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.545585 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.548687 (+ 3102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.193:55329: BlockingRecv error: recv got EOF from 127.8.221.193:55329 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 14:48:41.651837 17973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:41.652150 17973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:41.652200 17973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:41.656495 17973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:41.656567 17973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:41.656629 17973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:48:41.662391 17973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:38909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.194.keytab
--principal=oryx/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:45799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=45047
--tserver_master_addrs=127.8.221.254:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:41.663722 17973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:41.665019 17973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:41.676944 17983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:41.677067 17973 server_base.cc:1034] running on GCE node
W20260620 14:48:41.676930 17980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:41.676929 17981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:41.677865 17973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:41.678930 17973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:41.680130 17973 hybrid_clock.cc:648] HybridClock initialized: now 1781966921680101 us; error 41 us; skew 500 ppm
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966921, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:41.683477 17973 init.cc:377] Logged in from keytab as oryx/127.8.221.194@KRBTEST.COM (short username oryx)
I20260620 14:48:41.684680 17973 webserver.cc:460] Webserver started at http://127.8.221.194:45047/ using document root <none> and password file <none>
I20260620 14:48:41.685441 17973 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:41.685559 17973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:41.690599 17973 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:48:41.693478 17990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:41.695406 17973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:41.695578 17973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "df8b98e31b484377a4d831b16d4d1918"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-65tz"
server_key: "53bce856ee7c85bbf9406f6465c9a844"
server_key_iv: "d91d71ba7f268f49206630d9b2331f70"
server_key_version: "encryptionkey@0"
I20260620 14:48:41.696334 17973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:41.724061 17973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:41.727835 17973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:41.728094 17973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:41.728860 17973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:41.729959 17973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:41.730039 17973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:41.730121 17973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:41.730175 17973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:41.743045 17973 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:45799
I20260620 14:48:41.743069 18103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:45799 every 8 connection(s)
I20260620 14:48:41.744390 17973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:41.748972 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 17973
I20260620 14:48:41.749439 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:38895
--local_ip_for_outbound_sockets=127.8.221.195
--tserver_master_addrs=127.8.221.254:45155
--webserver_port=35747
--webserver_interface=127.8.221.195
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:38909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.221.195
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:48:41.759343 17975 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:41.746699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:41585 (local address 127.8.221.254:45155)
0620 14:48:41.746838 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:41.746843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:41.748541 (+ 1698us) server_negotiation.cc:407] Connection header received
0620 14:48:41.749979 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:41.749986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:41.750063 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:41.750200 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:41.752170 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.752828 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.754458 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.754696 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.754817 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.759148 (+ 4331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.194:41585: BlockingRecv error: recv got EOF from 127.8.221.194:41585 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:48:41.760215 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:41.747322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.194:41585)
0620 14:48:41.748331 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:41.748395 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:41.749577 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:41.750470 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:41.750478 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:41.751059 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:41.751784 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.751807 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.752983 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.752987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.754326 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.754338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.754887 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.754891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.755095 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.755920 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:41.755942 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:41.758976 (+ 3034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":202,"threads_started":1}
W20260620 14:48:41.760939 18104 heartbeater.cc:644] Failed to heartbeat to 127.8.221.254:45155 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.221.254:45155: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:41.768678 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:41.761646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.194:56633)
0620 14:48:41.761914 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:41.761931 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:41.762032 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:41.762352 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:41.762355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:41.762564 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:41.762726 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.762733 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.763678 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.763681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.764642 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.764651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.765191 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.765195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.765310 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.766061 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:41.766077 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:41.768491 (+ 2414us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174}
W20260620 14:48:41.768942 17975 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:41.761787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:56633 (local address 127.8.221.254:45155)
0620 14:48:41.761976 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:41.761980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:41.761998 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:48:41.762152 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:41.762155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:41.762206 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:41.762297 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:41.762856 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.763552 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.764789 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.765060 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.765171 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.768769 (+ 3598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.194:56633: BlockingRecv error: recv got EOF from 127.8.221.194:56633 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:41.776526 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:41.769696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.194:32911)
0620 14:48:41.769904 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:41.769918 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:41.769992 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:41.770623 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:41.770626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:41.770802 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:41.770929 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.770936 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.772073 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.772076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.772734 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.772743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.773319 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.773322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.773397 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.774060 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:41.774073 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:41.776377 (+ 2304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136}
W20260620 14:48:41.776852 18104 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:48:41.776863 17975 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:41.770130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:32911 (local address 127.8.221.254:45155)
0620 14:48:41.770318 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:41.770324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:41.770344 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:48:41.770396 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:41.770401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:41.770455 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:41.770548 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:41.771072 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.771935 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.772867 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.773184 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.773300 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.776558 (+ 3258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.194:32911: BlockingRecv error: recv got EOF from 127.8.221.194:32911 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 14:48:41.875130 18107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:41.875399 18107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:41.875442 18107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:41.879621 18107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:41.879686 18107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:41.879751 18107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:48:41.885120 18107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:38909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/krb5kdc/oryx_127.8.221.195.keytab
--principal=oryx/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:38895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=35747
--tserver_master_addrs=127.8.221.254:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:41.886281 18107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:41.887281 18107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:41.897768 18115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:41.897743 18113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:41.897979 18112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:41.898228 18107 server_base.cc:1034] running on GCE node
I20260620 14:48:41.898775 18107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:41.899483 18107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:41.900698 18107 hybrid_clock.cc:648] HybridClock initialized: now 1781966921900698 us; error 72 us; skew 500 ppm
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966921, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:41.904022 18107 init.cc:377] Logged in from keytab as oryx/127.8.221.195@KRBTEST.COM (short username oryx)
I20260620 14:48:41.904985 18107 webserver.cc:460] Webserver started at http://127.8.221.195:35747/ using document root <none> and password file <none>
I20260620 14:48:41.905602 18107 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:41.905681 18107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:41.910203 18107 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:41.912657 18122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:41.914228 18107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:41.914382 18107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "feb5d38eeb364cafa1a0921a602d07b4"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-65tz"
server_key: "20b9df7d4f9cae25483f534297e71e35"
server_key_iv: "5703c5d44e269e155b20524958f2848c"
server_key_version: "encryptionkey@0"
I20260620 14:48:41.914973 18107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:41.928301 18107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:41.931495 18107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:41.931718 18107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:41.932325 18107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:41.933185 18107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:41.933259 18107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:41.933328 18107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:41.933378 18107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:41.943706 18107 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:38895
I20260620 14:48:41.943784 18235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:38895 every 8 connection(s)
I20260620 14:48:41.944828 18107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:41.946933 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 18107
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:48:41.957769 17975 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:41.948816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:32875 (local address 127.8.221.254:45155)
0620 14:48:41.948991 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:41.948997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:41.949018 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:48:41.950163 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:41.950165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:41.950202 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:41.950258 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:41.951760 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.952295 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.953553 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.953773 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.953913 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.957632 (+ 3719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.195:32875: BlockingRecv error: recv got EOF from 127.8.221.195:32875 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 14:48:41.958761 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:41.949108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.8.221.254:45155)
0620 14:48:41.949346 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:41.949350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:41.949360 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:48:41.949399 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:41.949402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:41.949445 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:41.949538 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:41.950651 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.951207 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.952286 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.952543 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.952661 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.954358 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:41.954363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:41.954378 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:41.954405 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:41.956299 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:41.956836 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:41.956849 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:41.956854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:41.956966 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:41.957266 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:41.957271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:41.957272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:41.957721 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:41.957962 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:41.958345 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 14:48:41.958580 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
I20260620 14:48:41.958899 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:41.947793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.195:32875)
0620 14:48:41.948642 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:41.948729 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:41.949917 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:41.950506 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:41.950516 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:41.950924 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:41.951594 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.951614 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.952474 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.952478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.953409 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.953421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.953999 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.954005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.954246 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.954865 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:41.954890 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:41.957491 (+ 2601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":117,"threads_started":1}
W20260620 14:48:41.959801 18236 heartbeater.cc:644] Failed to heartbeat to 127.8.221.254:45155 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.221.254:45155: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:41.965958 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:41.960316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.195:38089)
0620 14:48:41.960471 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:41.960489 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:41.960591 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:41.960842 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:41.960846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:41.960985 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:41.961089 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.961095 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.962055 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.962058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.962719 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.962727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.963168 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.963171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.963241 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.963800 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:41.963814 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:41.965778 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 14:48:41.966162 18240 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:41.960389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:38089 (local address 127.8.221.254:45155)
0620 14:48:41.960488 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:41.960492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:41.960543 (+ 51us) server_negotiation.cc:407] Connection header received
0620 14:48:41.960682 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:41.960684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:41.960723 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:41.960790 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:41.961208 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.961945 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.962832 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.963049 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.963133 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.966010 (+ 2877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.195:38089: BlockingRecv error: recv got EOF from 127.8.221.195:38089 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:41 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:41.971840 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:41.966556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.195:50625)
0620 14:48:41.966655 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:41.966671 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:41.966762 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:41.966989 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:41.966992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:41.967153 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:41.967233 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.967238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.967935 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.967941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.968508 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.968515 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.969028 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.969031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.969136 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.969685 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:41.969703 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:41.971676 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 14:48:41.971916 18240 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:41.966618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:50625 (local address 127.8.221.254:45155)
0620 14:48:41.966775 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:41.966781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:41.966799 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:48:41.966847 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:41.966850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:41.966885 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:41.966973 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:41.967341 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.967833 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.968618 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.968923 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.969028 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.971799 (+ 2771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.195:50625: BlockingRecv error: recv got EOF from 127.8.221.195:50625 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 14:48:41.972136 18236 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 14:48:42 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:42 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:42.555612 18241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.549873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.193:56391)
0620 14:48:42.550159 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.550173 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:42.550270 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:42.550545 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:42.550547 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:42.550723 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:42.550823 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.550828 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.551701 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.551704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.552341 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.552350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.552815 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.552817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.552878 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.553462 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:42.553474 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:42.555419 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":100,"threads_started":1}
W20260620 14:48:42.555863 18242 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:42.549985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:56391 (local address 127.8.221.254:45155)
0620 14:48:42.550287 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.550291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.550305 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:48:42.550350 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.550353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.550405 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.550504 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.550965 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.551564 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.552475 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.552708 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.552802 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.555621 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.193:56391: BlockingRecv error: recv got EOF from 127.8.221.193:56391 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":102,"threads_started":1}
Jun 20 14:48:42 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:42 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:42.784286 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.777704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.194:51351)
0620 14:48:42.778219 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.778240 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:42.778369 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:42.778692 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:42.778696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:42.778929 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:42.779034 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.779041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.779871 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.779877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.780521 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.780536 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.781028 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.781032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.781121 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.781729 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:42.781748 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:42.784042 (+ 2294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":151,"threads_started":1}
W20260620 14:48:42.784598 18242 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:42.777827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:51351 (local address 127.8.221.254:45155)
0620 14:48:42.778000 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.778005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.778309 (+ 304us) server_negotiation.cc:407] Connection header received
0620 14:48:42.778499 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.778503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.778566 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.778664 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.779165 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.779753 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.780673 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.780907 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.781022 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.784439 (+ 3417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.194:51351: BlockingRecv error: recv got EOF from 127.8.221.194:51351 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 14:48:42 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:42 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:42.979452 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.972877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.195:51235)
0620 14:48:42.973211 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:42.973232 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:42.973419 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:42.973738 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:42.973741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:42.974054 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:42.974170 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.974177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.975024 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.975030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.975658 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:42.975665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.976141 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.976145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:42.976223 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.976837 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:42.976851 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:42.979204 (+ 2353us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":94,"threads_started":1}
W20260620 14:48:42.979489 18242 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:42.973039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:51235 (local address 127.8.221.254:45155)
0620 14:48:42.973213 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.973220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.973347 (+ 127us) server_negotiation.cc:407] Connection header received
0620 14:48:42.973546 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.973549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.973614 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.973709 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.974317 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.974909 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.975788 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.976019 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.976133 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.979369 (+ 3236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.195:51235: BlockingRecv error: recv got EOF from 127.8.221.195:51235 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 14:48:43 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:43 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:43.563750 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:43.556843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.193:54799)
0620 14:48:43.557232 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:43.557255 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:43.557419 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:43.557938 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:43.557942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:43.558217 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:43.558383 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:43.558392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:43.559415 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:43.559418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:43.560041 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:43.560048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:43.560651 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:43.560656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:43.560752 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:43.561381 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:43.561392 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:43.563599 (+ 2207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":128,"threads_started":1}
W20260620 14:48:43.563963 18246 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:43.556832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:54799 (local address 127.8.221.254:45155)
0620 14:48:43.557159 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:43.557163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:43.557360 (+ 197us) server_negotiation.cc:407] Connection header received
0620 14:48:43.557600 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:43.557604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:43.557676 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:43.557787 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:43.558513 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:43.559306 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:43.560158 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:43.560387 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:43.560610 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:43.563782 (+ 3172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.193:54799: BlockingRecv error: recv got EOF from 127.8.221.193:54799 (error 108)
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":157,"threads_started":1}
Jun 20 14:48:43 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:43 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:43.801529 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:43.785560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.194:60995)
0620 14:48:43.786008 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:43.786029 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:43.786222 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:43.786532 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:43.786542 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:43.786841 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:43.786959 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:43.786966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:43.787787 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:43.787793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:43.788390 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:43.788398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:43.788901 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:43.788905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:43.788979 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:43.798227 (+ 9248us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:43.798259 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:43.801255 (+ 2996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":106,"threads_started":1}
W20260620 14:48:43.801623 18246 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:43.785729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:60995 (local address 127.8.221.254:45155)
0620 14:48:43.785960 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:43.785966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:43.786148 (+ 182us) server_negotiation.cc:407] Connection header received
0620 14:48:43.786353 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:43.786361 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:43.786411 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:43.786507 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:43.787085 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:43.787671 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:43.788509 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:43.788746 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:43.788895 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:43.801484 (+ 12589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.194:60995: BlockingRecv error: recv got EOF from 127.8.221.194:60995 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 20 14:48:43 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:43 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:43.987092 18248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:43.980499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.195:53819)
0620 14:48:43.981005 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:43.981021 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:43.981191 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:43.981481 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:43.981485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:43.981691 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:43.981786 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:43.981842 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:43.982636 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:43.982639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:43.983199 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:43.983206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:43.983663 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:43.983667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:43.983749 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:43.984389 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:43.984401 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:43.986895 (+ 2494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":142,"threads_started":1}
W20260620 14:48:43.987237 18246 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:43.980731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:53819 (local address 127.8.221.254:45155)
0620 14:48:43.980952 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:43.980957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:43.981126 (+ 169us) server_negotiation.cc:407] Connection header received
0620 14:48:43.981309 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:43.981318 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:43.981363 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:43.981453 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:43.981967 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:43.982508 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:43.983317 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:43.983545 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:43.983656 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:43.987097 (+ 3441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.195:53819: BlockingRecv error: recv got EOF from 127.8.221.195:53819 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Jun 20 14:48:44 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:44 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:44.571192 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:44.564758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.193:60765)
0620 14:48:44.565131 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:44.565152 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:44.565274 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:44.565648 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:44.565650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:44.565884 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:44.565992 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:44.565998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:44.566912 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:44.566915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:44.567490 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:44.567497 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:44.567987 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:44.567990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:44.568051 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:44.568635 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:44.568646 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:44.571068 (+ 2422us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":104,"threads_started":1}
W20260620 14:48:44.571471 18250 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:44.564996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:60765 (local address 127.8.221.254:45155)
0620 14:48:44.565327 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:44.565331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:44.565347 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:48:44.565434 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:44.565438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:44.565504 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:44.565598 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:44.566106 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:44.566728 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:44.567610 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:44.567871 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:44.567964 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:44.571269 (+ 3305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.193:60765: BlockingRecv error: recv got EOF from 127.8.221.193:60765 (error 108)
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":93,"threads_started":1}
Jun 20 14:48:44 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:44 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:44.809415 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:44.802612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.194:49637)
0620 14:48:44.802976 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:44.802993 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:44.803103 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:44.803419 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:44.803422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:44.803608 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:44.803718 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:44.803724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:44.804574 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:44.804578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:44.805249 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:44.805257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:44.805936 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:44.805941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:44.806012 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:44.806695 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:44.806708 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:44.809203 (+ 2495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":95,"threads_started":1}
W20260620 14:48:44.809485 18250 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:44.802725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:49637 (local address 127.8.221.254:45155)
0620 14:48:44.803053 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:44.803063 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:44.803088 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:48:44.803221 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:44.803224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:44.803282 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:44.803368 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:44.803836 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:44.804435 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:44.805395 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:44.805670 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:44.805770 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:44.809362 (+ 3592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.194:49637: BlockingRecv error: recv got EOF from 127.8.221.194:49637 (error 108)
Metrics: {"server-negotiator.queue_time_us":186}
Jun 20 14:48:44 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:44 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:44.994621 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:44.988160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.195:43005)
0620 14:48:44.988596 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:44.988616 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:44.988792 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:44.989135 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:44.989138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:44.989399 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:44.989515 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:44.989524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:44.990480 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:44.990485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:44.991109 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:44.991117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:44.991658 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:44.991661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:44.991737 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:44.992322 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:44.992333 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:44.994414 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":170,"threads_started":1}
W20260620 14:48:44.994690 18250 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:44.988372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:43005 (local address 127.8.221.254:45155)
0620 14:48:44.988590 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:44.988596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:44.988724 (+ 128us) server_negotiation.cc:407] Connection header received
0620 14:48:44.988954 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:44.988957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:44.989016 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:44.989112 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:44.989619 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:44.990347 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:44.991237 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:44.991540 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:44.991653 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:44.994564 (+ 2911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.195:43005: BlockingRecv error: recv got EOF from 127.8.221.195:43005 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 20 14:48:45 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:45 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:45.579363 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:45.572171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.193:51571)
0620 14:48:45.572587 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:45.572606 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:45.572761 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:45.573143 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:45.573145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:45.573375 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:45.573508 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:45.573513 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:45.574474 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:45.574478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:45.575175 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:45.575185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:45.575718 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:45.575721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:45.575791 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:45.576513 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:45.576531 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:45.579145 (+ 2614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":134,"threads_started":1}
W20260620 14:48:45.579470 18254 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:45.572389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:51571 (local address 127.8.221.254:45155)
0620 14:48:45.572625 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:45.572634 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:45.572694 (+ 60us) server_negotiation.cc:407] Connection header received
0620 14:48:45.572938 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:45.572940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:45.572993 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:45.573089 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:45.573653 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:45.574318 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:45.575329 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:45.575566 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:45.575662 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:45.579321 (+ 3659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.193:51571: BlockingRecv error: recv got EOF from 127.8.221.193:51571 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
Jun 20 14:48:45 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:45 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:45.816987 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:45.810599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.194:58603)
0620 14:48:45.811006 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:45.811024 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:45.811167 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:45.811473 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:45.811476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:45.811700 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:45.811791 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:45.811796 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:45.812561 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:45.812566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:45.813131 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:45.813139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:45.813612 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:45.813615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:45.813698 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:45.814394 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:45.814405 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:45.816836 (+ 2431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":102,"threads_started":1}
W20260620 14:48:45.817088 18254 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:45.810611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:58603 (local address 127.8.221.254:45155)
0620 14:48:45.810781 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:45.810785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:45.811111 (+ 326us) server_negotiation.cc:407] Connection header received
0620 14:48:45.811306 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:45.811309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:45.811359 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:45.811451 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:45.811909 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:45.812449 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:45.813251 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:45.813495 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:45.813609 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:45.816984 (+ 3375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.194:58603: BlockingRecv error: recv got EOF from 127.8.221.194:58603 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 14:48:45 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:45 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:46.002013 18256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:45.995563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.195:59233)
0620 14:48:45.995930 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:45.995944 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:45.996067 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:45.996355 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:45.996357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:45.996600 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:45.996718 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:45.996724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:45.997584 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:45.997587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:45.998223 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:45.998231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:45.998865 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:45.998870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:45.998991 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:45.999564 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:45.999576 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:46.001866 (+ 2290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":112,"threads_started":1}
W20260620 14:48:46.002282 18254 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:45.995672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:59233 (local address 127.8.221.254:45155)
0620 14:48:45.995827 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:45.995831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:45.996081 (+ 250us) server_negotiation.cc:407] Connection header received
0620 14:48:45.996147 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:45.996150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:45.996212 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:45.996321 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:45.996866 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:45.997434 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:45.998346 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:45.998590 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:45.998826 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:46.002102 (+ 3276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.195:59233: BlockingRecv error: recv got EOF from 127.8.221.195:59233 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 14:48:46 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:46 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:46.587074 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:46.580465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.193:45561)
0620 14:48:46.580839 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:46.580859 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:46.580985 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:46.581271 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:46.581273 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:46.581486 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:46.581644 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:46.581649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.582581 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.582584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:46.583182 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:46.583190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.583641 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.583643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:46.583708 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:46.584323 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:46.584335 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:46.586869 (+ 2534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":127,"threads_started":1}
W20260620 14:48:46.587245 18258 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:46.580560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:45561 (local address 127.8.221.254:45155)
0620 14:48:46.580835 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:46.580839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:46.580952 (+ 113us) server_negotiation.cc:407] Connection header received
0620 14:48:46.581085 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:46.581087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:46.581131 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:46.581242 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:46.581771 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.582457 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.583305 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.583527 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.583614 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:46.587060 (+ 3446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.193:45561: BlockingRecv error: recv got EOF from 127.8.221.193:45561 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":96,"threads_started":1}
Jun 20 14:48:46 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:46 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:46.824329 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:46.818112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.194:39835)
0620 14:48:46.818387 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:46.818401 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:46.818491 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:46.818866 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:46.818869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:46.819162 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:46.819284 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:46.819291 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.820251 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.820254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:46.820855 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:46.820863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.821342 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.821345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:46.821404 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:46.822016 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:46.822028 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:46.824171 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":99,"threads_started":1}
W20260620 14:48:46.824469 18258 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:46.818305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:39835 (local address 127.8.221.254:45155)
0620 14:48:46.818481 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:46.818487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:46.818507 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:48:46.818595 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:46.818598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:46.818664 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:46.818775 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:46.819404 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.820138 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.820976 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.821226 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.821327 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:46.824337 (+ 3010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.194:39835: BlockingRecv error: recv got EOF from 127.8.221.194:39835 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 14:48:47 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:47 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:47.009135 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.002982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.195:47201)
0620 14:48:47.003338 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.003353 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:47.003496 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:47.003803 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:47.003805 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:47.004076 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:47.004179 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.004185 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.005020 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.005026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.005602 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.005609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.006153 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.006156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.006234 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.006933 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:47.006945 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:47.008926 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":111,"threads_started":1}
W20260620 14:48:47.009176 18258 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:47.003082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:47201 (local address 127.8.221.254:45155)
0620 14:48:47.003279 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.003284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.003442 (+ 158us) server_negotiation.cc:407] Connection header received
0620 14:48:47.003620 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.003624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.003687 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.003780 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:47.004316 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.004904 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.005723 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.006035 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.006149 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.009062 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.195:47201: BlockingRecv error: recv got EOF from 127.8.221.195:47201 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 14:48:47 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:47 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:47.596087 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.588341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.193:45015)
0620 14:48:47.588737 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.588755 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:47.588869 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:47.589257 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:47.589260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:47.589467 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:47.589620 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.589626 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.590606 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.590609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.591294 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.591305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.591908 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.591911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.591993 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.592666 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:47.592678 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:47.595950 (+ 3272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":102,"threads_started":1}
W20260620 14:48:47.596393 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:47.588402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:45015 (local address 127.8.221.254:45155)
0620 14:48:47.588804 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.588809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.588825 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:48:47.588987 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.588989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.589046 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.589146 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:47.589766 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.590409 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.591469 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.591751 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.591858 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.596167 (+ 4309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.193:45015: BlockingRecv error: recv got EOF from 127.8.221.193:45015 (error 108)
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":157,"threads_started":1}
Jun 20 14:48:47 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:47 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:47.831352 18263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.825404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.194:51377)
0620 14:48:47.825729 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.825747 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:47.825920 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:47.826299 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:47.826301 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:47.826543 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:47.826683 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.826694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.827516 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.827519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.828121 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.828129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.828587 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.828589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.828673 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.829258 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:47.829268 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:47.831223 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
W20260620 14:48:47.831560 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:47.825520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:51377 (local address 127.8.221.254:45155)
0620 14:48:47.825691 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.825697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.825857 (+ 160us) server_negotiation.cc:407] Connection header received
0620 14:48:47.826077 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.826080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.826140 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.826266 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:47.826816 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.827398 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.828235 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.828481 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.828567 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.831383 (+ 2816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.194:51377: BlockingRecv error: recv got EOF from 127.8.221.194:51377 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 14:48:48 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:48 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:48.017016 18264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.010339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.195:38433)
0620 14:48:48.010712 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.010729 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:48.010873 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:48.011262 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:48.011265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:48.011500 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:48.011617 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:48.011623 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.012422 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.012425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:48.013057 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:48.013064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.013550 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.013553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:48.013617 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.014260 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:48.014279 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:48.016866 (+ 2587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":99,"threads_started":1}
W20260620 14:48:48.017211 18262 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:48.010547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:38433 (local address 127.8.221.254:45155)
0620 14:48:48.010756 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:48.010761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:48.010829 (+ 68us) server_negotiation.cc:407] Connection header received
0620 14:48:48.011034 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:48.011037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:48.011101 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:48.011201 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:48.011758 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.012301 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.013179 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.013407 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.013510 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.017029 (+ 3519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.195:38433: BlockingRecv error: recv got EOF from 127.8.221.195:38433 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Jun 20 14:48:48 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:48 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:48.604048 18265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.597134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.193:56629)
0620 14:48:48.597474 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.597488 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:48.597590 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:48.597903 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:48.597905 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:48.598093 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:48.598193 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:48.598198 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.599248 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.599261 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:48.600059 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:48.600070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.600668 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.600671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:48.600735 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.601484 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:48.601518 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:48.603899 (+ 2381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":100,"threads_started":1}
W20260620 14:48:48.604251 18266 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:48.597206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:56629 (local address 127.8.221.254:45155)
0620 14:48:48.597562 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:48.597566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:48.597584 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:48:48.597678 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:48.597681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:48.597734 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:48.597883 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:48.598339 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.599065 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.600216 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.600539 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.600646 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.604083 (+ 3437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.193:56629: BlockingRecv error: recv got EOF from 127.8.221.193:56629 (error 108)
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":161,"threads_started":1}
Jun 20 14:48:48 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:48 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:48.843451 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.832406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.194:44547)
0620 14:48:48.832785 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.832800 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:48.832924 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:48.833205 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:48.833207 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:48.833463 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:48.833581 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:48.833586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.834395 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.834398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:48.835144 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:48.835159 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.835807 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.835813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:48.835931 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.841029 (+ 5098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:48.841042 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:48.843252 (+ 2210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":112,"threads_started":1}
W20260620 14:48:48.843518 18266 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:48.832539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:44547 (local address 127.8.221.254:45155)
0620 14:48:48.832718 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:48.832722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:48.832888 (+ 166us) server_negotiation.cc:407] Connection header received
0620 14:48:48.833021 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:48.833023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:48.833076 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:48.833167 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:48.833697 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.834255 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.835293 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.835627 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.835780 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.843401 (+ 7621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.194:44547: BlockingRecv error: recv got EOF from 127.8.221.194:44547 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 14:48:49 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:49 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:49.025091 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:49.018122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.195:59801)
0620 14:48:49.018571 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:49.018587 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:49.018743 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:49.019265 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:49.019267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:49.019591 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:49.019750 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:49.019761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.020710 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.020713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:49.021376 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:49.021385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.021941 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.021945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:49.022016 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:49.022603 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:49.022618 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:49.024853 (+ 2235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":84,"threads_started":1}
W20260620 14:48:49.025143 18266 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:49.018353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:59801 (local address 127.8.221.254:45155)
0620 14:48:49.018541 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:49.018547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:49.018681 (+ 134us) server_negotiation.cc:407] Connection header received
0620 14:48:49.018952 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:49.018959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:49.019036 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:49.019158 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:49.019905 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.020566 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.021516 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.021742 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.021870 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:49.025023 (+ 3153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.195:59801: BlockingRecv error: recv got EOF from 127.8.221.195:59801 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 14:48:49 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:49 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:49.611371 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:49.605082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.193:52729)
0620 14:48:49.605458 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:49.605474 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:49.605594 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:49.605897 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:49.605899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:49.606118 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:49.606238 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:49.606243 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.607117 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.607120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:49.607893 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:49.607902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.608420 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.608424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:49.608502 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:49.609068 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:49.609079 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:49.611243 (+ 2164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":147,"threads_started":1}
W20260620 14:48:49.611644 18270 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:49.605329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:52729 (local address 127.8.221.254:45155)
0620 14:48:49.605620 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:49.605625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:49.605659 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:48:49.605704 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:49.605706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:49.605753 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:49.605867 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:49.606357 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.606994 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.608025 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.608291 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.608415 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:49.611475 (+ 3060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.193:52729: BlockingRecv error: recv got EOF from 127.8.221.193:52729 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":90,"threads_started":1}
Jun 20 14:48:49 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:49 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:49.851500 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:49.844531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.194:53807)
0620 14:48:49.844811 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:49.844825 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:49.844935 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:49.845232 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:49.845234 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:49.845411 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:49.845515 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:49.845521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.846666 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.846669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:49.847404 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:49.847413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.847898 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.847904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:49.848030 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:49.848692 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:49.848705 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:49.851360 (+ 2655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":97,"threads_started":1}
W20260620 14:48:49.851780 18270 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:49.844603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:53807 (local address 127.8.221.254:45155)
0620 14:48:49.844788 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:49.844792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:49.844907 (+ 115us) server_negotiation.cc:407] Connection header received
0620 14:48:49.845040 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:49.845043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:49.845098 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:49.845195 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:49.845625 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.846540 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.847522 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:49.847758 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:49.847853 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:49.851580 (+ 3727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.194:53807: BlockingRecv error: recv got EOF from 127.8.221.194:53807 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 14:48:50 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:50 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:50.033052 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:50.026195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.195:59219)
0620 14:48:50.026570 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:50.026585 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:50.026715 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:50.027017 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:50.027019 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:50.027272 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:50.027393 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:50.027399 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:50.028180 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:50.028182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:50.028828 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:50.028836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:50.029372 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:50.029376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:50.029455 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:50.030104 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:50.030117 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:50.032903 (+ 2786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":125,"threads_started":1}
W20260620 14:48:50.033231 18270 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:50.026332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:59219 (local address 127.8.221.254:45155)
0620 14:48:50.026494 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:50.026498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:50.026676 (+ 178us) server_negotiation.cc:407] Connection header received
0620 14:48:50.026811 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:50.026813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:50.026867 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:50.026972 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:50.027524 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:50.028072 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:50.028969 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:50.029197 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:50.029295 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:50.033088 (+ 3793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.195:59219: BlockingRecv error: recv got EOF from 127.8.221.195:59219 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 14:48:50 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:50 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:50.618640 18273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:50.612492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.193:39763)
0620 14:48:50.612949 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:50.612965 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:50.613076 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:50.613344 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:50.613347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:50.613532 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:50.613637 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:50.613642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:50.614462 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:50.614465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:50.615046 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:50.615065 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:50.615556 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:50.615559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:50.615619 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:50.616193 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:50.616205 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:50.618401 (+ 2196us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":135,"threads_started":1}
W20260620 14:48:50.618692 18274 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:50.612625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:39763 (local address 127.8.221.254:45155)
0620 14:48:50.612959 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:50.612964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:50.613026 (+ 62us) server_negotiation.cc:407] Connection header received
0620 14:48:50.613157 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:50.613160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:50.613213 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:50.613303 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:50.613745 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:50.614350 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:50.615195 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:50.615443 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:50.615538 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:50.618532 (+ 2994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.193:39763: BlockingRecv error: recv got EOF from 127.8.221.193:39763 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":76,"threads_started":1}
Jun 20 14:48:50 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:50 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:50.858911 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:50.852517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.194:49795)
0620 14:48:50.852870 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:50.852885 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:50.853047 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:50.853342 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:50.853345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:50.853577 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:50.853685 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:50.853691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:50.854532 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:50.854538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:50.855163 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:50.855171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:50.855646 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:50.855650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:50.855734 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:50.856358 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:50.856370 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:50.858694 (+ 2324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":105,"threads_started":1}
W20260620 14:48:50.858953 18274 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:50.852640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:49795 (local address 127.8.221.254:45155)
0620 14:48:50.852823 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:50.852838 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:50.852977 (+ 139us) server_negotiation.cc:407] Connection header received
0620 14:48:50.853164 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:50.853169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:50.853229 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:50.853323 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:50.853840 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:50.854419 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:50.855292 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:50.855515 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:50.855632 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:50.858830 (+ 3198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.194:49795: BlockingRecv error: recv got EOF from 127.8.221.194:49795 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 14:48:51 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:51 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:51.039753 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:51.034029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.195:46327)
0620 14:48:51.034335 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:51.034348 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:51.034454 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:51.034794 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:51.034797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:51.034980 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:51.035101 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:51.035107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:51.035936 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:51.035939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:51.036532 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:51.036540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:51.036972 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:51.036975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:51.037033 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:51.037629 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:51.037640 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:51.039615 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":121,"threads_started":1}
W20260620 14:48:51.039994 18274 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:51.034173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:46327 (local address 127.8.221.254:45155)
0620 14:48:51.034363 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:51.034367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:51.034428 (+ 61us) server_negotiation.cc:407] Connection header received
0620 14:48:51.034600 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:51.034603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:51.034668 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:51.034755 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:51.035225 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:51.035825 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:51.036651 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:51.036869 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:51.036953 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:51.039779 (+ 2826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.195:46327: BlockingRecv error: recv got EOF from 127.8.221.195:46327 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 20 14:48:51 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:51 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:51.626574 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:51.619997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.193:38093)
0620 14:48:51.620319 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:51.620334 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:51.620467 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:51.620864 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:51.620866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:51.621033 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:51.621132 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:51.621136 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:51.622099 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:51.622102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:51.622718 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:51.622726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:51.623248 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:51.623251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:51.623340 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:51.624001 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:51.624013 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:51.626442 (+ 2429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":102,"threads_started":1}
W20260620 14:48:51.626878 18278 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:51.620086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:38093 (local address 127.8.221.254:45155)
0620 14:48:51.620393 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:51.620418 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:51.620435 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:48:51.620641 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:51.620645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:51.620703 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:51.620820 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:51.621273 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:51.621968 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:51.622840 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:51.623071 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:51.623166 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:51.626703 (+ 3537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.193:38093: BlockingRecv error: recv got EOF from 127.8.221.193:38093 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":124,"threads_started":1}
Jun 20 14:48:51 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:51 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:51.866660 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:51.860023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.194:60693)
0620 14:48:51.860441 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:51.860461 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:51.860641 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:51.860970 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:51.860973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:51.861305 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:51.861444 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:51.861451 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:51.862363 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:51.862368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:51.862961 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:51.862969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:51.863440 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:51.863443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:51.863522 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:51.864120 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:51.864132 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:51.866528 (+ 2396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":100,"threads_started":1}
W20260620 14:48:51.866873 18278 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:51.860182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:60693 (local address 127.8.221.254:45155)
0620 14:48:51.860418 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:51.860424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:51.860577 (+ 153us) server_negotiation.cc:407] Connection header received
0620 14:48:51.860782 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:51.860785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:51.860837 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:51.860945 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:51.861575 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:51.862245 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:51.863082 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:51.863315 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:51.863429 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:51.866683 (+ 3254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.194:60693: BlockingRecv error: recv got EOF from 127.8.221.194:60693 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 20 14:48:52 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:52 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:52.047240 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:52.040746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.195:37121)
0620 14:48:52.041150 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:52.041172 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:52.041303 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:52.041639 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:52.041642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:52.041902 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:52.042007 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:52.042013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:52.042840 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:52.042845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:52.043428 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:52.043435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:52.043888 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:52.043892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:52.043967 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:52.044602 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:52.044613 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:52.047031 (+ 2418us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":117,"threads_started":1}
W20260620 14:48:52.047390 18278 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:52.041009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:37121 (local address 127.8.221.254:45155)
0620 14:48:52.041250 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:52.041256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:52.041272 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:48:52.041429 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:52.041433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:52.041495 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:52.041590 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:52.042134 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:52.042715 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:52.043544 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:52.043770 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:52.043883 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:52.047214 (+ 3331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.195:37121: BlockingRecv error: recv got EOF from 127.8.221.195:37121 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Jun 20 14:48:52 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:52 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:52.633298 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:52.627413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.193:52143)
0620 14:48:52.627780 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:52.627797 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:52.627923 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:52.628250 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:52.628252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:52.628426 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:52.628528 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:52.628533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:52.629357 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:52.629361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:52.629957 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:52.629966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:52.630421 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:52.630423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:52.630487 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:52.631299 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:52.631310 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:52.633146 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":133,"threads_started":1}
W20260620 14:48:52.633550 18282 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:52.627564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:52143 (local address 127.8.221.254:45155)
0620 14:48:52.627851 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:52.627856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:52.627870 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:48:52.628024 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:52.628026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:52.628077 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:52.628176 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:52.628634 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:52.629223 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:52.630072 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:52.630311 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:52.630391 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:52.633337 (+ 2946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.193:52143: BlockingRecv error: recv got EOF from 127.8.221.193:52143 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":111,"threads_started":1}
Jun 20 14:48:52 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:52 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:52.874079 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:52.867624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.194:58897)
0620 14:48:52.867989 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:52.868005 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:52.868129 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:52.868568 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:52.868572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:52.868767 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:52.868939 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:52.868946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:52.869860 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:52.869864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:52.870617 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:52.870627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:52.871117 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:52.871119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:52.871217 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:52.871832 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:52.871853 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:52.873958 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":83,"threads_started":1}
W20260620 14:48:52.874392 18282 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:52.867702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:58897 (local address 127.8.221.254:45155)
0620 14:48:52.867890 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:52.867898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:52.868068 (+ 170us) server_negotiation.cc:407] Connection header received
0620 14:48:52.868290 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:52.868293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:52.868384 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:52.868511 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:52.869067 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:52.869711 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:52.870725 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:52.871010 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:52.871100 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:52.874223 (+ 3123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.194:58897: BlockingRecv error: recv got EOF from 127.8.221.194:58897 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 14:48:53 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:53 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:53.054443 18284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:53.048286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.195:54907)
0620 14:48:53.048608 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:53.048625 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:53.048785 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:53.049082 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:53.049085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:53.049320 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:53.049416 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:53.049422 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:53.050366 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:53.050372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:53.051015 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:53.051022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:53.051517 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:53.051521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:53.051610 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:53.052223 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:53.052237 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:53.054257 (+ 2020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":93,"threads_started":1}
W20260620 14:48:53.054512 18282 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:53.048371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:54907 (local address 127.8.221.254:45155)
0620 14:48:53.048551 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:53.048556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:53.048737 (+ 181us) server_negotiation.cc:407] Connection header received
0620 14:48:53.048903 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:53.048906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:53.048968 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:53.049064 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:53.049548 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:53.050246 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:53.051146 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:53.051398 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:53.051515 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:53.054386 (+ 2871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.195:54907: BlockingRecv error: recv got EOF from 127.8.221.195:54907 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 14:48:53 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:53 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:53.640358 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:53.634236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.193:36773)
0620 14:48:53.634568 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:53.634584 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:53.634708 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:53.634997 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:53.635000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:53.635214 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:53.635336 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:53.635342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:53.636200 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:53.636203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:53.636805 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:53.636812 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:53.637305 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:53.637308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:53.637372 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:53.638029 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:53.638040 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:53.640227 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":89,"threads_started":1}
W20260620 14:48:53.640538 18286 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:53.634330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:36773 (local address 127.8.221.254:45155)
0620 14:48:53.634588 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:53.634592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:53.634708 (+ 116us) server_negotiation.cc:407] Connection header received
0620 14:48:53.634782 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:53.634785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:53.634859 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:53.634958 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:53.635480 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:53.636043 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:53.636929 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:53.637193 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:53.637282 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:53.640363 (+ 3081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.193:36773: BlockingRecv error: recv got EOF from 127.8.221.193:36773 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":68,"threads_started":1}
Jun 20 14:48:53 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:53 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:53.884603 18287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:53.875049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.194:35439)
0620 14:48:53.875459 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:53.875479 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:53.875662 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:53.875970 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:53.875973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:53.876224 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:53.876331 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:53.876337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:53.877198 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:53.877204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:53.877786 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:53.877825 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:53.878298 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:53.878302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:53.878381 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:53.882330 (+ 3949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:53.882342 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:53.884372 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":109,"threads_started":1}
W20260620 14:48:53.884650 18286 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:53.875237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:35439 (local address 127.8.221.254:45155)
0620 14:48:53.875436 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:53.875442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:53.875586 (+ 144us) server_negotiation.cc:407] Connection header received
0620 14:48:53.875780 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:53.875783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:53.875841 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:53.875947 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:53.876457 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:53.877079 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:53.877951 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:53.878176 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:53.878290 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:53.884533 (+ 6243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.194:35439: BlockingRecv error: recv got EOF from 127.8.221.194:35439 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 14:48:54 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:54 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:54.061533 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:54.055366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.195:54035)
0620 14:48:54.055639 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:54.055653 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:54.055758 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:54.056039 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:54.056041 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:54.056313 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:54.056452 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:54.056458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:54.057341 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:54.057344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:54.057961 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:54.057969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:54.058536 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:54.058539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:54.058604 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:54.059145 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:54.059157 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:54.061303 (+ 2146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":98,"threads_started":1}
W20260620 14:48:54.061563 18286 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:54.055493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:54035 (local address 127.8.221.254:45155)
0620 14:48:54.055643 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:54.055648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:54.055732 (+ 84us) server_negotiation.cc:407] Connection header received
0620 14:48:54.055860 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:54.055863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:54.055919 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:54.056001 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:54.056575 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:54.057167 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:54.058149 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:54.058380 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:54.058493 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:54.061447 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.195:54035: BlockingRecv error: recv got EOF from 127.8.221.195:54035 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 14:48:54 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:54 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:54.648284 18289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:54.641368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.193:37133)
0620 14:48:54.641701 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:54.641717 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:54.641866 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:54.642214 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:54.642216 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:54.642457 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:54.642579 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:54.642585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:54.643526 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:54.643528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:54.644162 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:54.644170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:54.644700 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:54.644702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:54.644767 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:54.645366 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:54.645386 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:54.648057 (+ 2671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":91,"threads_started":1}
W20260620 14:48:54.648478 18290 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:54.641470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:37133 (local address 127.8.221.254:45155)
0620 14:48:54.641762 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:54.641766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:54.641782 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:48:54.641980 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:54.641982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:54.642033 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:54.642130 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:54.642753 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:54.643392 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:54.644303 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:54.644574 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:54.644683 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:54.648250 (+ 3567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.193:37133: BlockingRecv error: recv got EOF from 127.8.221.193:37133 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":91,"threads_started":1}
Jun 20 14:48:54 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:54 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:54.892443 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:54.885732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.194:40591)
0620 14:48:54.886146 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:54.886162 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:54.886288 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:54.886566 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:54.886568 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:54.886782 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:54.886900 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:54.886905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:54.887682 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:54.887688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:54.888549 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:54.888558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:54.889279 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:54.889283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:54.889408 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:54.890077 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:54.890089 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:54.892281 (+ 2192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":206,"threads_started":1}
W20260620 14:48:54.892617 18290 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:54.885915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:40591 (local address 127.8.221.254:45155)
0620 14:48:54.886066 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:54.886070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:54.886244 (+ 174us) server_negotiation.cc:407] Connection header received
0620 14:48:54.886381 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:54.886383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:54.886431 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:54.886524 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:54.887002 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:54.887553 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:54.888734 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:54.889137 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:54.889270 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:54.892462 (+ 3192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.194:40591: BlockingRecv error: recv got EOF from 127.8.221.194:40591 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 14:48:55 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:55 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:55.068585 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:55.062656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.195:36351)
0620 14:48:55.062976 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:55.062991 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:55.063128 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:55.063418 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:55.063421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:55.063638 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:55.063726 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:55.063731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:55.064469 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:55.064475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:55.065054 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:55.065062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:55.065575 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:55.065579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:55.065659 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:55.066295 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:55.066307 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:55.068452 (+ 2145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":112,"threads_started":1}
W20260620 14:48:55.068899 18290 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:55.062715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:36351 (local address 127.8.221.254:45155)
0620 14:48:55.062889 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:55.062895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:55.063067 (+ 172us) server_negotiation.cc:407] Connection header received
0620 14:48:55.063257 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:55.063262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:55.063314 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:55.063400 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:55.063841 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:55.064360 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:55.065172 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:55.065392 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:55.065575 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:55.068691 (+ 3116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.195:36351: BlockingRecv error: recv got EOF from 127.8.221.195:36351 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 14:48:55 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:55 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:48:55.655813 18294 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:55.649424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:52227 (local address 127.8.221.254:45155)
0620 14:48:55.649890 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:55.649894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:55.649909 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:48:55.650056 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:55.650059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:55.650107 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:55.650193 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:55.650674 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:55.651210 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:55.652114 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:55.652469 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:55.652590 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:55.655692 (+ 3102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.193:52227: BlockingRecv error: recv got EOF from 127.8.221.193:52227 (error 108)
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":101,"threads_started":1}
I20260620 14:48:55.655802 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:55.649330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.193:52227)
0620 14:48:55.649670 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:55.649692 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:55.649947 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:55.650237 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:55.650239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:55.650452 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:55.650571 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:55.650576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:55.651317 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:55.651320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:55.652000 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:55.652008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:55.652592 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:55.652595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:55.652704 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:55.653408 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:55.653419 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:55.655570 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":74,"threads_started":1}
Jun 20 14:48:55 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:55 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:48:55.899586 18294 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:55.893568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:37111 (local address 127.8.221.254:45155)
0620 14:48:55.893753 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:55.893757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:55.893946 (+ 189us) server_negotiation.cc:407] Connection header received
0620 14:48:55.894033 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:55.894036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:55.894089 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:55.894172 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:55.894668 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:55.895230 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:55.896110 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:55.896322 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:55.896407 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:55.899488 (+ 3081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.194:37111: BlockingRecv error: recv got EOF from 127.8.221.194:37111 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:48:55.899588 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:55.893442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.194:37111)
0620 14:48:55.893819 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:55.893834 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:55.893952 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:55.894199 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:55.894201 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:55.894416 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:55.894531 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:55.894536 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:55.895345 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:55.895348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:55.895990 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:55.895998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:55.896427 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:55.896429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:55.896492 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:55.897172 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:55.897206 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:55.899371 (+ 2165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":140,"threads_started":1}
Jun 20 14:48:56 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:56 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.195@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:56.075562 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:56.069575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.195:56209)
0620 14:48:56.070038 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:56.070053 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:56.070197 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:56.070522 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:56.070525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:56.070727 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:56.070813 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:56.070818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:56.071583 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:56.071590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:56.072238 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:56.072246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:56.072730 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:56.072734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:56.072812 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:56.073416 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:56.073438 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:56.075395 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":201,"threads_started":1}
W20260620 14:48:56.075762 18294 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:56.069714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:56209 (local address 127.8.221.254:45155)
0620 14:48:56.069975 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:56.069980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:56.070130 (+ 150us) server_negotiation.cc:407] Connection header received
0620 14:48:56.070322 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:56.070327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:56.070409 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:56.070501 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:56.070937 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:56.071465 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:56.072378 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:56.072612 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:56.072721 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:56.075559 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.195:56209: BlockingRecv error: recv got EOF from 127.8.221.195:56209 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 14:48:56 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:56 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.193@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:56.663013 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:56.657000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.193:60785)
0620 14:48:56.657365 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:56.657382 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:56.657479 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:56.657872 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:56.657875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:56.658105 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:56.658212 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:56.658217 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:56.659056 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:56.659059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:56.659637 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:56.659644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:56.660065 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:56.660067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:56.660127 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:56.660729 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:56.660744 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:56.662879 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
W20260620 14:48:56.663256 18298 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:56.657086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:60785 (local address 127.8.221.254:45155)
0620 14:48:56.657422 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:56.657426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:56.657444 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:48:56.657593 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:56.657596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:56.657649 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:56.657756 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:56.658328 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:56.658924 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:56.659745 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:56.659963 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:56.660040 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:56.663060 (+ 3020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.193:60785: BlockingRecv error: recv got EOF from 127.8.221.193:60785 (error 108)
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":106,"threads_started":1}
Jun 20 14:48:56 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:56 dist-test-slave-65tz krb5kdc[17240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.221.194@KRBTEST.COM for oryx/127.8.221.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:56.907135 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:56.900851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:45155 (local address 127.8.221.194:57167)
0620 14:48:56.901203 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:56.901224 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:56.901346 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:56.901662 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:56.901665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:56.901961 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:56.902068 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:56.902075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:56.902956 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:56.902962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:56.903564 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:56.903571 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:56.904051 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:56.904055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:56.904131 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:56.904751 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:56.904764 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:56.906932 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.221.254:45155: Server oryx/127.8.221.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":107,"threads_started":1}
W20260620 14:48:56.907239 18298 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:56.900955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:57167 (local address 127.8.221.254:45155)
0620 14:48:56.901274 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:56.901281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:56.901297 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:48:56.901484 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:56.901487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:56.901547 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:56.901642 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:56.902197 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:56.902838 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:56.903694 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:56.903931 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:56.904045 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:56.907075 (+ 3030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.221.194:57167: BlockingRecv error: recv got EOF from 127.8.221.194:57167 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:48:56.947369 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 17823
I20260620 14:48:56.954375 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 17973
I20260620 14:48:56.960752 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 18107
I20260620 14:48:56.967947 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 17753
2026-06-20T14:48:56Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18247 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T14:48:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:56Z Disabled control of system clock
I20260620 14:48:57.005465 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:40681
--webserver_interface=127.8.221.254
--webserver_port=0
--builtin_ntp_servers=127.8.221.212:35571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:40681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:48:57.135670 18303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:57.136057 18303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:57.136125 18303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:57.140306 18303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:57.140405 18303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:57.140436 18303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:57.140475 18303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:57.140509 18303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:57.145388 18303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:40681
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:40681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:57.146767 18303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:57.148442 18303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:57.158337 18311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:57.158360 18308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:57.158337 18309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:57.158603 18303 server_base.cc:1034] running on GCE node
I20260620 14:48:57.159333 18303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:57.160665 18303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:57.161909 18303 hybrid_clock.cc:648] HybridClock initialized: now 1781966937161872 us; error 60 us; skew 500 ppm
I20260620 14:48:57.164170 18303 webserver.cc:460] Webserver started at http://127.8.221.254:34659/ using document root <none> and password file <none>
I20260620 14:48:57.164853 18303 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:57.164952 18303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:57.165234 18303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:57.167722 18303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "e3d285c3aed34c48836034bee04eb2fa"
format_stamp: "Formatted at 2026-06-20 14:48:57 on dist-test-slave-65tz"
server_key: "e6b4fe8efe7a259ca480b932414aad41"
server_key_iv: "2ce0f566a714cbaaa0f0727d0e963883"
server_key_version: "encryptionkey@0"
I20260620 14:48:57.168413 18303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "e3d285c3aed34c48836034bee04eb2fa"
format_stamp: "Formatted at 2026-06-20 14:48:57 on dist-test-slave-65tz"
server_key: "e6b4fe8efe7a259ca480b932414aad41"
server_key_iv: "2ce0f566a714cbaaa0f0727d0e963883"
server_key_version: "encryptionkey@0"
I20260620 14:48:57.173158 18303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:48:57.176357 18317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:57.178138 18303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:57.178308 18303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "e3d285c3aed34c48836034bee04eb2fa"
format_stamp: "Formatted at 2026-06-20 14:48:57 on dist-test-slave-65tz"
server_key: "e6b4fe8efe7a259ca480b932414aad41"
server_key_iv: "2ce0f566a714cbaaa0f0727d0e963883"
server_key_version: "encryptionkey@0"
I20260620 14:48:57.178450 18303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:57.193440 18303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:57.194434 18303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:57.194697 18303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:57.203733 18303 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:40681
I20260620 14:48:57.203785 18369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:40681 every 8 connection(s)
I20260620 14:48:57.204963 18303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:57.208514 18370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:57.212628 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 18303
I20260620 14:48:57.212759 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:48:57.213073 9079 external_mini_cluster.cc:1428] Setting key cc9ed4a4d4500fb68eaa93186b60876b
I20260620 14:48:57.215018 18370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa: Bootstrap starting.
I20260620 14:48:57.217240 18370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:57.218173 18370 log.cc:824] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:57.220274 18370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa: No bootstrap required, opened a new log
I20260620 14:48:57.224193 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:57.214959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.8.221.254:40681)
0620 14:48:57.215860 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:57.215880 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:57.215926 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:48:57.216724 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:57.216744 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:57.217239 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:57.217622 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:57.218118 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:57.219218 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:57.220434 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:57.220976 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:57.221108 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:57.222077 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:57.222082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:57.222095 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:57.222550 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:57.222614 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:57.222634 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:57.222731 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:57.222915 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 14:48:57.223269 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":224,"threads_started":1}
I20260620 14:48:57.224428 18370 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d285c3aed34c48836034bee04eb2fa" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 40681 } }
I20260620 14:48:57.224711 18370 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:57.224773 18370 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3d285c3aed34c48836034bee04eb2fa, State: Initialized, Role: FOLLOWER
I20260620 14:48:57.225265 18370 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d285c3aed34c48836034bee04eb2fa" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 40681 } }
I20260620 14:48:57.225417 18370 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:57.225487 18370 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:57.225593 18370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:57.226533 18370 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d285c3aed34c48836034bee04eb2fa" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 40681 } }
I20260620 14:48:57.226890 18370 leader_election.cc:302] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3d285c3aed34c48836034bee04eb2fa; no voters:
I20260620 14:48:57.227214 18370 leader_election.cc:288] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:57.227376 18375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:57.227672 18375 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa [term 1 LEADER]: Becoming Leader. State: Replica: e3d285c3aed34c48836034bee04eb2fa, State: Running, Role: LEADER
I20260620 14:48:57.228067 18375 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d285c3aed34c48836034bee04eb2fa" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 40681 } }
I20260620 14:48:57.228489 18370 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:57.230013 18376 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3d285c3aed34c48836034bee04eb2fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d285c3aed34c48836034bee04eb2fa" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 40681 } } }
I20260620 14:48:57.230142 18376 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:57.230072 18377 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3d285c3aed34c48836034bee04eb2fa. Latest consensus state: current_term: 1 leader_uuid: "e3d285c3aed34c48836034bee04eb2fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3d285c3aed34c48836034bee04eb2fa" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 40681 } } }
I20260620 14:48:57.230211 18377 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:57.230578 18386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:57.233893 18386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:57.240175 18386 catalog_manager.cc:1347] Generated new cluster ID: e28f3b27452d42b6b1ad36b6e8c8061c
I20260620 14:48:57.240262 18386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:57.248558 18386 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:57.250092 18386 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:57.268447 18386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3d285c3aed34c48836034bee04eb2fa: Generated new TSK 0
I20260620 14:48:57.269635 18386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:48:57.280752 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:40681
--builtin_ntp_servers=127.8.221.212:35571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 14:48:57.408941 18394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:57.409337 18394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:57.409415 18394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:57.413751 18394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:57.413865 18394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:57.413961 18394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:48:57.419044 18394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:40681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:57.420488 18394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:57.422233 18394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:57.433533 18400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:57.433570 18399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:57.433576 18402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:57.433841 18394 server_base.cc:1034] running on GCE node
I20260620 14:48:57.434407 18394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:57.435174 18394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:57.436381 18394 hybrid_clock.cc:648] HybridClock initialized: now 1781966937436350 us; error 52 us; skew 500 ppm
I20260620 14:48:57.438552 18394 webserver.cc:460] Webserver started at http://127.8.221.193:39055/ using document root <none> and password file <none>
I20260620 14:48:57.439196 18394 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:57.439284 18394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:57.439527 18394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:57.441689 18394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "f4cbbb1238c64d3dae7a21057f125f46"
format_stamp: "Formatted at 2026-06-20 14:48:57 on dist-test-slave-65tz"
server_key: "5cf7ebc5f7a495d8fd77291e3760ef52"
server_key_iv: "57ecfd30cd5531db23107b9eb70a3f61"
server_key_version: "encryptionkey@0"
I20260620 14:48:57.442312 18394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "f4cbbb1238c64d3dae7a21057f125f46"
format_stamp: "Formatted at 2026-06-20 14:48:57 on dist-test-slave-65tz"
server_key: "5cf7ebc5f7a495d8fd77291e3760ef52"
server_key_iv: "57ecfd30cd5531db23107b9eb70a3f61"
server_key_version: "encryptionkey@0"
I20260620 14:48:57.446329 18394 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 14:48:57.449160 18408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:57.450688 18394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:57.450826 18394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "f4cbbb1238c64d3dae7a21057f125f46"
format_stamp: "Formatted at 2026-06-20 14:48:57 on dist-test-slave-65tz"
server_key: "5cf7ebc5f7a495d8fd77291e3760ef52"
server_key_iv: "57ecfd30cd5531db23107b9eb70a3f61"
server_key_version: "encryptionkey@0"
I20260620 14:48:57.450943 18394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:57.461671 18394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:57.462344 18394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:57.462539 18394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:57.463091 18394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:57.464005 18394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:57.464074 18394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:57.464143 18394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:57.464190 18394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:57.475699 18394 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:37083
I20260620 14:48:57.475780 18521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:37083 every 8 connection(s)
I20260620 14:48:57.477154 18394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:57.478452 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 18394
I20260620 14:48:57.478564 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:48:57.478891 9079 external_mini_cluster.cc:1428] Setting key 76ddc1efdd8ebff2d75d03341d4ac578
I20260620 14:48:57.483385 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:40681
--builtin_ntp_servers=127.8.221.212:35571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:48:57.489857 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:57.482584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:58961 (local address 127.8.221.254:40681)
0620 14:48:57.482742 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:57.482748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:57.482767 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:48:57.484074 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:57.484082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:57.484167 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:57.484297 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:57.485428 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:57.486199 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:57.487460 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:57.487711 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:57.487833 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:57.489280 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:57.489284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:57.489286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:57.489300 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:57.489318 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:57.489327 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:57.489403 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:57.489595 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 14:48:57.489663 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:48:57.490808 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:57.480689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:40681 (local address 127.8.221.193:58961)
0620 14:48:57.481882 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:57.481974 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:57.483806 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:57.484464 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:57.484471 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:57.484561 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:57.485237 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:57.485265 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:57.486339 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:57.486347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:57.487301 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:57.487311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:57.487876 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:57.487880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:57.488068 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:57.488881 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:57.488882 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:57.488928 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:57.489004 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:57.489431 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:57.489440 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:57.489446 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:48:57.489581 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 14:48:57.489896 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":200,"threads_started":1}
I20260620 14:48:57.492249 18522 heartbeater.cc:347] Connected to a master server at 127.8.221.254:40681
I20260620 14:48:57.492504 18522 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:57.493191 18522 heartbeater.cc:511] Master 127.8.221.254:40681 requested a full tablet report, sending...
I20260620 14:48:57.495358 18334 ts_manager.cc:194] Registered new tserver with Master: f4cbbb1238c64d3dae7a21057f125f46 (127.8.221.193:37083)
I20260620 14:48:57.497519 18334 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.221.193:58961
W20260620 14:48:57.624605 18525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:57.624953 18525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:57.624998 18525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:57.629817 18525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:57.629868 18525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:57.629930 18525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:48:57.635216 18525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:40681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:57.636564 18525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:57.638273 18525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:57.649305 18531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:57.649304 18533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:57.649369 18530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:57.649585 18525 server_base.cc:1034] running on GCE node
I20260620 14:48:57.650136 18525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:57.650926 18525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:57.652143 18525 hybrid_clock.cc:648] HybridClock initialized: now 1781966937652118 us; error 53 us; skew 500 ppm
I20260620 14:48:57.654438 18525 webserver.cc:460] Webserver started at http://127.8.221.194:33135/ using document root <none> and password file <none>
I20260620 14:48:57.655230 18525 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:57.655337 18525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:57.655616 18525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:57.658200 18525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "4ec0e3f6ef4d45c889e928aad8257e01"
format_stamp: "Formatted at 2026-06-20 14:48:57 on dist-test-slave-65tz"
server_key: "df143f26874ac9b4b10204bf939e833e"
server_key_iv: "1dcd50ee35cff2e74fbbedf2b1d9bb4a"
server_key_version: "encryptionkey@0"
I20260620 14:48:57.658878 18525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "4ec0e3f6ef4d45c889e928aad8257e01"
format_stamp: "Formatted at 2026-06-20 14:48:57 on dist-test-slave-65tz"
server_key: "df143f26874ac9b4b10204bf939e833e"
server_key_iv: "1dcd50ee35cff2e74fbbedf2b1d9bb4a"
server_key_version: "encryptionkey@0"
I20260620 14:48:57.663436 18525 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:48:57.666666 18539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:57.668329 18525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:48:57.668501 18525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "4ec0e3f6ef4d45c889e928aad8257e01"
format_stamp: "Formatted at 2026-06-20 14:48:57 on dist-test-slave-65tz"
server_key: "df143f26874ac9b4b10204bf939e833e"
server_key_iv: "1dcd50ee35cff2e74fbbedf2b1d9bb4a"
server_key_version: "encryptionkey@0"
I20260620 14:48:57.668624 18525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:57.682950 18525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:57.683861 18525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:57.684090 18525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:57.684805 18525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:57.686040 18525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:57.686129 18525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:57.686244 18525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:57.686290 18525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:57.699514 18525 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:35601
I20260620 14:48:57.699549 18652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:35601 every 8 connection(s)
I20260620 14:48:57.700641 18525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:57.701655 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 18525
I20260620 14:48:57.701834 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:57.702180 9079 external_mini_cluster.cc:1428] Setting key f53e150cad60e39e9b282e95b9b4a914
I20260620 14:48:57.704797 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:40681
--builtin_ntp_servers=127.8.221.212:35571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:48:57.713609 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:57.702886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:57531 (local address 127.8.221.254:40681)
0620 14:48:57.703064 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:57.703068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:57.704526 (+ 1458us) server_negotiation.cc:407] Connection header received
0620 14:48:57.706931 (+ 2405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:57.706936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:57.707015 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:57.707139 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:57.708497 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:57.709065 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:57.711168 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:57.711391 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:57.711504 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:57.713082 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:57.713085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:57.713087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:57.713101 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:57.713120 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:57.713128 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:57.713191 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:57.713424 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 14:48:57.713486 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:48:57.715339 18655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:57.703393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:40681 (local address 127.8.221.194:57531)
0620 14:48:57.704315 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:57.704378 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:57.706567 (+ 2189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:57.707399 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:57.707409 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:57.707516 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:57.708295 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:57.708326 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:57.709227 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:57.709234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:57.711007 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:57.711026 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:57.711595 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:57.711601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:57.711875 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:57.712679 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:57.712683 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:57.712755 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:57.712873 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:57.713226 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:57.713237 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:57.713247 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 14:48:57.713428 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 14:48:57.713882 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":171,"threads_started":1}
I20260620 14:48:57.716722 18653 heartbeater.cc:347] Connected to a master server at 127.8.221.254:40681
I20260620 14:48:57.717080 18653 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:57.718066 18653 heartbeater.cc:511] Master 127.8.221.254:40681 requested a full tablet report, sending...
I20260620 14:48:57.719632 18334 ts_manager.cc:194] Registered new tserver with Master: 4ec0e3f6ef4d45c889e928aad8257e01 (127.8.221.194:35601)
I20260620 14:48:57.720304 18334 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.221.194:57531
W20260620 14:48:57.840317 18656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:57.840651 18656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:57.840729 18656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:57.845048 18656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:57.845139 18656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:57.845237 18656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:48:57.850214 18656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:40681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:57.851557 18656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:57.853314 18656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:57.865057 18662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:57.865059 18661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:57.865065 18664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:57.865463 18656 server_base.cc:1034] running on GCE node
I20260620 14:48:57.866022 18656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:57.866863 18656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:57.868099 18656 hybrid_clock.cc:648] HybridClock initialized: now 1781966937868051 us; error 62 us; skew 500 ppm
I20260620 14:48:57.870440 18656 webserver.cc:460] Webserver started at http://127.8.221.195:41843/ using document root <none> and password file <none>
I20260620 14:48:57.871188 18656 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:57.871296 18656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:57.871640 18656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:57.874524 18656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "d7f49d65192e461cab5c2b2c64dca91d"
format_stamp: "Formatted at 2026-06-20 14:48:57 on dist-test-slave-65tz"
server_key: "2e42a231bda30a4dbe9bf1615f60920a"
server_key_iv: "2f208ca8b16c79d60ddaed797ca63bbd"
server_key_version: "encryptionkey@0"
I20260620 14:48:57.875205 18656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "d7f49d65192e461cab5c2b2c64dca91d"
format_stamp: "Formatted at 2026-06-20 14:48:57 on dist-test-slave-65tz"
server_key: "2e42a231bda30a4dbe9bf1615f60920a"
server_key_iv: "2f208ca8b16c79d60ddaed797ca63bbd"
server_key_version: "encryptionkey@0"
I20260620 14:48:57.880053 18656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:48:57.883608 18670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:57.885246 18656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:48:57.885443 18656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "d7f49d65192e461cab5c2b2c64dca91d"
format_stamp: "Formatted at 2026-06-20 14:48:57 on dist-test-slave-65tz"
server_key: "2e42a231bda30a4dbe9bf1615f60920a"
server_key_iv: "2f208ca8b16c79d60ddaed797ca63bbd"
server_key_version: "encryptionkey@0"
I20260620 14:48:57.885601 18656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:57.904019 18656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:57.904742 18656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:57.904950 18656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:57.905570 18656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:57.906646 18656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:57.906723 18656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:57.906805 18656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:57.906852 18656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:57.916610 18656 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:34679
I20260620 14:48:57.916754 18783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:34679 every 8 connection(s)
I20260620 14:48:57.917589 18656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:57.923974 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 18656
I20260620 14:48:57.924131 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:57.924445 9079 external_mini_cluster.cc:1428] Setting key 0468881b9789206794b1db4b754ab820
I20260620 14:48:57.928633 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:57.919851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:43331 (local address 127.8.221.254:40681)
0620 14:48:57.920033 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:57.920037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:57.921176 (+ 1139us) server_negotiation.cc:407] Connection header received
0620 14:48:57.922852 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:57.922856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:57.922919 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:57.923034 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:57.924301 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:57.924904 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:57.926458 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:57.926736 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:57.926848 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:57.928136 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:57.928139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:57.928141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:57.928153 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:57.928171 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:57.928179 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:57.928242 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:57.928460 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 14:48:57.928516 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:48:57.929755 18786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:57.920224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:40681 (local address 127.8.221.195:43331)
0620 14:48:57.920981 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:57.921029 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:57.922610 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:57.923225 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:57.923233 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:57.923325 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:57.924039 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:57.924062 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:57.925097 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:57.925100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:57.926330 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:57.926341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:57.926904 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:57.926908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:57.927068 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:57.927782 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:57.927784 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:57.927831 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:57.927907 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:57.928262 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:57.928269 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:57.928274 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:48:57.928387 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:48:57.928682 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":117,"threads_started":1}
I20260620 14:48:57.931044 18784 heartbeater.cc:347] Connected to a master server at 127.8.221.254:40681
I20260620 14:48:57.931361 18784 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:57.931969 18784 heartbeater.cc:511] Master 127.8.221.254:40681 requested a full tablet report, sending...
I20260620 14:48:57.933087 18334 ts_manager.cc:194] Registered new tserver with Master: d7f49d65192e461cab5c2b2c64dca91d (127.8.221.195:34679)
I20260620 14:48:57.933653 18334 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.221.195:43331
I20260620 14:48:57.938798 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:48:57.942044 18373 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:57.940866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60836 (local address 127.8.221.254:40681)
0620 14:48:57.941028 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:57.941035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:57.941289 (+ 254us) server_negotiation.cc:407] Connection header received
0620 14:48:57.941453 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:57.941458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:57.941509 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:57.941598 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:57.941884 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60836: BlockingRecv error: recv got EOF from 127.0.0.1:60836 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:48:57.942291 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 18394
I20260620 14:48:57.948868 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 18525
I20260620 14:48:57.955278 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 18656
I20260620 14:48:57.961848 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 18303
2026-06-20T14:48:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (988 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T14:48:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:57Z Disabled control of system clock
I20260620 14:48:57.994436 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:40045
--webserver_interface=127.8.221.254
--webserver_port=0
--builtin_ntp_servers=127.8.221.212:36855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:40045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:48:58.111187 18795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:58.111457 18795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:58.111495 18795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:58.115602 18795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:58.115664 18795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:58.115679 18795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:58.115696 18795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:58.115712 18795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:58.120556 18795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:36855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:40045
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:40045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:58.121591 18795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:58.123040 18795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:58.131791 18803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:58.131912 18801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:58.131913 18800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:58.132438 18795 server_base.cc:1034] running on GCE node
I20260620 14:48:58.132968 18795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:58.134141 18795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:58.135318 18795 hybrid_clock.cc:648] HybridClock initialized: now 1781966938135260 us; error 78 us; skew 500 ppm
I20260620 14:48:58.137238 18795 webserver.cc:460] Webserver started at http://127.8.221.254:38499/ using document root <none> and password file <none>
I20260620 14:48:58.137910 18795 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:58.137982 18795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:58.138197 18795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:58.140223 18795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "35e7726baa484a3186258fba5c1bf7e3"
format_stamp: "Formatted at 2026-06-20 14:48:58 on dist-test-slave-65tz"
server_key: "c0b0c950bec56d821ec228a1ebeec721"
server_key_iv: "998dc0c604b7347620c916029fcb4a51"
server_key_version: "encryptionkey@0"
I20260620 14:48:58.140806 18795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "35e7726baa484a3186258fba5c1bf7e3"
format_stamp: "Formatted at 2026-06-20 14:48:58 on dist-test-slave-65tz"
server_key: "c0b0c950bec56d821ec228a1ebeec721"
server_key_iv: "998dc0c604b7347620c916029fcb4a51"
server_key_version: "encryptionkey@0"
I20260620 14:48:58.144896 18795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:48:58.147723 18809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:58.149123 18795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:58.149281 18795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "35e7726baa484a3186258fba5c1bf7e3"
format_stamp: "Formatted at 2026-06-20 14:48:58 on dist-test-slave-65tz"
server_key: "c0b0c950bec56d821ec228a1ebeec721"
server_key_iv: "998dc0c604b7347620c916029fcb4a51"
server_key_version: "encryptionkey@0"
I20260620 14:48:58.149408 18795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:58.154340 18795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:58.154985 18795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:58.155175 18795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:58.162472 18795 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:40045
I20260620 14:48:58.162493 18861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:40045 every 8 connection(s)
I20260620 14:48:58.163461 18795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:58.168651 18862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:58.170802 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 18795
I20260620 14:48:58.170938 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:48:58.171334 9079 external_mini_cluster.cc:1428] Setting key ea9ae37a94ef47a834e8028bc1c4ed0b
I20260620 14:48:58.179191 18862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3: Bootstrap starting.
I20260620 14:48:58.182340 18865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:58.174972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32868 (local address 127.8.221.254:40045)
0620 14:48:58.176056 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:58.176090 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:58.176166 (+ 76us) server_negotiation.cc:407] Connection header received
0620 14:48:58.177628 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:58.177658 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:58.178257 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:58.178929 (+ 672us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:58.180405 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:58.180411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:58.180427 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:58.180437 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:58.180506 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:58.180529 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:58.180685 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:58.180816 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 14:48:58.181148 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":186,"threads_started":1}
I20260620 14:48:58.182874 18862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:58.183866 18862 log.cc:824] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:58.186741 18862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3: No bootstrap required, opened a new log
I20260620 14:48:58.192598 18862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e7726baa484a3186258fba5c1bf7e3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 40045 } }
I20260620 14:48:58.192950 18862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:58.193015 18862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e7726baa484a3186258fba5c1bf7e3, State: Initialized, Role: FOLLOWER
I20260620 14:48:58.193517 18862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e7726baa484a3186258fba5c1bf7e3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 40045 } }
I20260620 14:48:58.193696 18862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:58.193768 18862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:58.193914 18862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:58.195318 18862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e7726baa484a3186258fba5c1bf7e3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 40045 } }
I20260620 14:48:58.195716 18862 leader_election.cc:302] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35e7726baa484a3186258fba5c1bf7e3; no voters:
I20260620 14:48:58.196069 18862 leader_election.cc:288] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:58.196273 18867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:58.197701 18862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:58.198038 18867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3 [term 1 LEADER]: Becoming Leader. State: Replica: 35e7726baa484a3186258fba5c1bf7e3, State: Running, Role: LEADER
I20260620 14:48:58.198495 18867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e7726baa484a3186258fba5c1bf7e3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 40045 } }
I20260620 14:48:58.199831 18867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35e7726baa484a3186258fba5c1bf7e3. Latest consensus state: current_term: 1 leader_uuid: "35e7726baa484a3186258fba5c1bf7e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e7726baa484a3186258fba5c1bf7e3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 40045 } } }
I20260620 14:48:58.199956 18867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:58.200275 18868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35e7726baa484a3186258fba5c1bf7e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e7726baa484a3186258fba5c1bf7e3" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 40045 } } }
I20260620 14:48:58.200366 18868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3 [sys.catalog]: This master's current role is: LEADER
W20260620 14:48:58.204584 18882 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:48:58.206317 18882 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:48:58.206431 18880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:58.207440 18880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:58.213181 18880 catalog_manager.cc:1347] Generated new cluster ID: 8b16fb4ed78047cf97b79f53a4d8511d
I20260620 14:48:58.213244 18880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:58.223081 18880 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:58.223400 18880 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:58.231855 18880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35e7726baa484a3186258fba5c1bf7e3: Generated new TSK 0
I20260620 14:48:58.232656 18880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:48:58.240042 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:40045
--builtin_ntp_servers=127.8.221.212:36855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:48:58.367880 18886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:58.368175 18886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:58.368248 18886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:58.372344 18886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:58.372426 18886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:58.372512 18886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:48:58.377678 18886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:36855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:40045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:58.378824 18886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:58.380200 18886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:58.390170 18892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:58.390193 18891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:58.390588 18886 server_base.cc:1034] running on GCE node
W20260620 14:48:58.390317 18894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:58.391079 18886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:58.391772 18886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:58.392970 18886 hybrid_clock.cc:648] HybridClock initialized: now 1781966938392941 us; error 41 us; skew 500 ppm
I20260620 14:48:58.395125 18886 webserver.cc:460] Webserver started at http://127.8.221.193:38817/ using document root <none> and password file <none>
I20260620 14:48:58.395761 18886 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:58.395859 18886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:58.396097 18886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:58.398393 18886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "5b40c685e9484bd78c9e2f7e6192d566"
format_stamp: "Formatted at 2026-06-20 14:48:58 on dist-test-slave-65tz"
server_key: "316af294b1e8ffb4ecfb57f79e01e3ec"
server_key_iv: "fb363c7c2db237203e94d73d612549c0"
server_key_version: "encryptionkey@0"
I20260620 14:48:58.399011 18886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "5b40c685e9484bd78c9e2f7e6192d566"
format_stamp: "Formatted at 2026-06-20 14:48:58 on dist-test-slave-65tz"
server_key: "316af294b1e8ffb4ecfb57f79e01e3ec"
server_key_iv: "fb363c7c2db237203e94d73d612549c0"
server_key_version: "encryptionkey@0"
I20260620 14:48:58.403301 18886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:48:58.406195 18900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:58.407846 18886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:48:58.408000 18886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "5b40c685e9484bd78c9e2f7e6192d566"
format_stamp: "Formatted at 2026-06-20 14:48:58 on dist-test-slave-65tz"
server_key: "316af294b1e8ffb4ecfb57f79e01e3ec"
server_key_iv: "fb363c7c2db237203e94d73d612549c0"
server_key_version: "encryptionkey@0"
I20260620 14:48:58.408123 18886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:58.412992 18886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:58.413586 18886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:58.413776 18886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:58.414399 18886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:58.415336 18886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:58.415407 18886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:58.415467 18886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:58.415519 18886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:58.425360 18886 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:46195
I20260620 14:48:58.425410 19013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:46195 every 8 connection(s)
I20260620 14:48:58.426378 18886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:58.428262 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 18886
I20260620 14:48:58.428404 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:48:58.428732 9079 external_mini_cluster.cc:1428] Setting key 1b40d8be9bc2d59ec6d17dddb42bc9c6
I20260620 14:48:58.431244 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:40045
--builtin_ntp_servers=127.8.221.212:36855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 14:48:58.434008 18865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:58.430023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:50971 (local address 127.8.221.254:40045)
0620 14:48:58.430176 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:58.430181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:58.430511 (+ 330us) server_negotiation.cc:407] Connection header received
0620 14:48:58.431998 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:58.432005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:58.432068 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:58.432178 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:58.433360 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:58.433367 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:58.433374 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:58.433396 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:58.433430 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:58.433444 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:58.433522 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:58.433775 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 14:48:58.433869 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:48:58.435134 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:58.429204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:40045 (local address 127.8.221.193:50971)
0620 14:48:58.430312 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:58.430375 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:58.431769 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:58.432340 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:58.432346 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:58.432433 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:58.433021 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:58.433022 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:58.433060 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:58.433150 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:58.433599 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:58.433609 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:58.433617 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:48:58.433850 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 14:48:58.434113 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":199,"threads_started":1}
I20260620 14:48:58.436468 19014 heartbeater.cc:347] Connected to a master server at 127.8.221.254:40045
I20260620 14:48:58.436707 19014 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:58.437139 19014 heartbeater.cc:511] Master 127.8.221.254:40045 requested a full tablet report, sending...
I20260620 14:48:58.439095 18822 ts_manager.cc:194] Registered new tserver with Master: 5b40c685e9484bd78c9e2f7e6192d566 (127.8.221.193:46195)
W20260620 14:48:58.560117 19017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:58.560413 19017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:58.560456 19017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:58.564514 19017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:58.564576 19017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:58.564633 19017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:48:58.570299 19017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:36855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:40045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:58.572557 19017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:58.574591 19017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:58.586191 19023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:58.586501 19017 server_base.cc:1034] running on GCE node
W20260620 14:48:58.586203 19025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:58.586813 19022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:58.587356 19017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:58.588174 19017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:58.589395 19017 hybrid_clock.cc:648] HybridClock initialized: now 1781966938589365 us; error 53 us; skew 500 ppm
I20260620 14:48:58.591643 19017 webserver.cc:460] Webserver started at http://127.8.221.194:33241/ using document root <none> and password file <none>
I20260620 14:48:58.592307 19017 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:58.592372 19017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:58.592566 19017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:58.595084 19017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "defe69f54d054a8ab54741170f30e8ae"
format_stamp: "Formatted at 2026-06-20 14:48:58 on dist-test-slave-65tz"
server_key: "b98a68965d1dbfdb2ae0da691da42508"
server_key_iv: "05d0731ef2496b7466d15f2ccdca0c13"
server_key_version: "encryptionkey@0"
I20260620 14:48:58.595693 19017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "defe69f54d054a8ab54741170f30e8ae"
format_stamp: "Formatted at 2026-06-20 14:48:58 on dist-test-slave-65tz"
server_key: "b98a68965d1dbfdb2ae0da691da42508"
server_key_iv: "05d0731ef2496b7466d15f2ccdca0c13"
server_key_version: "encryptionkey@0"
I20260620 14:48:58.600358 19017 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:48:58.603441 19031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:58.604943 19017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:48:58.605052 19017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "defe69f54d054a8ab54741170f30e8ae"
format_stamp: "Formatted at 2026-06-20 14:48:58 on dist-test-slave-65tz"
server_key: "b98a68965d1dbfdb2ae0da691da42508"
server_key_iv: "05d0731ef2496b7466d15f2ccdca0c13"
server_key_version: "encryptionkey@0"
I20260620 14:48:58.605154 19017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:58.610370 19017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:58.611012 19017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:58.611209 19017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:58.611892 19017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:58.613021 19017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:58.613098 19017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:58.613166 19017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:58.613210 19017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:58.624539 19017 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:41463
I20260620 14:48:58.624583 19144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:41463 every 8 connection(s)
I20260620 14:48:58.625654 19017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:58.630093 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 19017
I20260620 14:48:58.630267 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:58.630645 9079 external_mini_cluster.cc:1428] Setting key 93a042bc773795f100caf043378e0f22
I20260620 14:48:58.633222 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:40045
--builtin_ntp_servers=127.8.221.212:36855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 14:48:58.633522 18865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:58.627870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:53641 (local address 127.8.221.254:40045)
0620 14:48:58.628055 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:58.628059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:58.629350 (+ 1291us) server_negotiation.cc:407] Connection header received
0620 14:48:58.631328 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:58.631332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:58.631386 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:58.631501 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:58.632873 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:58.632876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:58.632879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:58.632892 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:58.632919 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:58.632929 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:58.633019 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:58.633255 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 14:48:58.633331 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:48:58.636206 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:58.628277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:40045 (local address 127.8.221.194:53641)
0620 14:48:58.629096 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:58.629182 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:58.631046 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:58.631695 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:58.631704 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:58.631803 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:58.632535 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:58.632537 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:58.632595 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:58.632694 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:58.633036 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:58.633047 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:58.633056 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 14:48:58.633293 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 14:48:58.633587 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":144,"threads_started":1}
I20260620 14:48:58.637511 19145 heartbeater.cc:347] Connected to a master server at 127.8.221.254:40045
I20260620 14:48:58.637877 19145 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:58.638386 19145 heartbeater.cc:511] Master 127.8.221.254:40045 requested a full tablet report, sending...
I20260620 14:48:58.639631 18822 ts_manager.cc:194] Registered new tserver with Master: defe69f54d054a8ab54741170f30e8ae (127.8.221.194:41463)
W20260620 14:48:58.766918 19148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:58.767231 19148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:58.767279 19148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:58.771919 19148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:58.771982 19148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:58.772047 19148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:48:58.777624 19148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:36855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:40045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:48:58.779448 19148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:58.781080 19148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:58.794114 19156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:58.794091 19154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:58.794091 19153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:58.794502 19148 server_base.cc:1034] running on GCE node
I20260620 14:48:58.794967 19148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:58.795679 19148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:58.796885 19148 hybrid_clock.cc:648] HybridClock initialized: now 1781966938796845 us; error 58 us; skew 500 ppm
I20260620 14:48:58.798939 19148 webserver.cc:460] Webserver started at http://127.8.221.195:43851/ using document root <none> and password file <none>
I20260620 14:48:58.799610 19148 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:58.799705 19148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:58.799968 19148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:58.802250 19148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "45067df17aa24417a4231ad63a066114"
format_stamp: "Formatted at 2026-06-20 14:48:58 on dist-test-slave-65tz"
server_key: "98b3e28af6f6504da5f899c2eca9ca5e"
server_key_iv: "4a162575c63b6473fbbb0381f21ea2e3"
server_key_version: "encryptionkey@0"
I20260620 14:48:58.802824 19148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "45067df17aa24417a4231ad63a066114"
format_stamp: "Formatted at 2026-06-20 14:48:58 on dist-test-slave-65tz"
server_key: "98b3e28af6f6504da5f899c2eca9ca5e"
server_key_iv: "4a162575c63b6473fbbb0381f21ea2e3"
server_key_version: "encryptionkey@0"
I20260620 14:48:58.806950 19148 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:58.810204 19162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:58.811825 19148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:58.812058 19148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "45067df17aa24417a4231ad63a066114"
format_stamp: "Formatted at 2026-06-20 14:48:58 on dist-test-slave-65tz"
server_key: "98b3e28af6f6504da5f899c2eca9ca5e"
server_key_iv: "4a162575c63b6473fbbb0381f21ea2e3"
server_key_version: "encryptionkey@0"
I20260620 14:48:58.812206 19148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:58.817198 19148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:58.817898 19148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:58.818085 19148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:58.818650 19148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:58.819594 19148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:58.819667 19148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:58.819727 19148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:58.819780 19148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:58.829844 19148 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:38571
I20260620 14:48:58.829870 19275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:38571 every 8 connection(s)
I20260620 14:48:58.830766 19148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:58.831482 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 19148
I20260620 14:48:58.831625 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:58.831985 9079 external_mini_cluster.cc:1428] Setting key b299c8a0dcdc7a678fd2b3e8c683e074
I20260620 14:48:58.839360 18865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:58.833289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:43529 (local address 127.8.221.254:40045)
0620 14:48:58.833480 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:58.833486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:58.835361 (+ 1875us) server_negotiation.cc:407] Connection header received
0620 14:48:58.837184 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:58.837187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:58.837244 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:58.837329 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:58.838789 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:58.838793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:58.838796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:58.838810 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:58.838840 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:58.838857 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:58.838971 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:58.839125 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 14:48:58.839183 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:48:58.840535 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:58.833896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:40045 (local address 127.8.221.195:43529)
0620 14:48:58.835121 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:58.835196 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:58.836928 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:58.837556 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:58.837565 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:58.837683 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:58.838480 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:58.838481 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:58.838539 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:58.838622 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:58.838961 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:58.838970 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:58.838978 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:48:58.839187 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 14:48:58.839528 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":191,"threads_started":1}
I20260620 14:48:58.842147 19276 heartbeater.cc:347] Connected to a master server at 127.8.221.254:40045
I20260620 14:48:58.842491 19276 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:58.842994 19276 heartbeater.cc:511] Master 127.8.221.254:40045 requested a full tablet report, sending...
I20260620 14:48:58.844369 18822 ts_manager.cc:194] Registered new tserver with Master: 45067df17aa24417a4231ad63a066114 (127.8.221.195:38571)
I20260620 14:48:58.848366 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:48:58.852312 18865 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:58.851153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.8.221.254:40045)
0620 14:48:58.851373 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:58.851389 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:58.851442 (+ 53us) server_negotiation.cc:407] Connection header received
0620 14:48:58.851624 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:58.851631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:58.851682 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:58.851818 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:58.852115 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32880: BlockingRecv error: recv got EOF from 127.0.0.1:32880 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 14:48:58.852535 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 18886
I20260620 14:48:58.860311 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 19017
I20260620 14:48:58.867112 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 19148
I20260620 14:48:58.873548 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 18795
2026-06-20T14:48:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (911 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:58 dist-test-slave-65tz krb5kdc[19286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:58 dist-test-slave-65tz krb5kdc[19286](info): set up 2 sockets
Jun 20 14:48:58 dist-test-slave-65tz krb5kdc[19286](info): commencing operation
krb5kdc: starting...
W20260620 14:48:59.943126 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:49:00 dist-test-slave-65tz krb5kdc[19286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:49:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:49:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:49:00.080746 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38881
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:38881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:49:00.210183 19299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:00.210598 19299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:00.210677 19299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:00.215574 19299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:49:00.215708 19299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:00.215746 19299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:00.215788 19299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:49:00.215823 19299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:49:00.221868 19299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:38881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:49:00.223317 19299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:00.224601 19299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:00.234680 19304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:00.234695 19307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:00.234680 19305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:00.235091 19299 server_base.cc:1034] running on GCE node
I20260620 14:49:00.235795 19299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:00.237028 19299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:00.238219 19299 hybrid_clock.cc:648] HybridClock initialized: now 1781966940238199 us; error 35 us; skew 500 ppm
Jun 20 14:49:00 dist-test-slave-65tz krb5kdc[19286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966940, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:00.241163 19299 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:49:00.242179 19299 webserver.cc:460] Webserver started at http://127.8.221.254:41619/ using document root <none> and password file <none>
I20260620 14:49:00.242743 19299 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:00.242805 19299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:00.242978 19299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:49:00.245026 19299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "ac2abe08d40e4147a3ae73ee88101d46"
format_stamp: "Formatted at 2026-06-20 14:49:00 on dist-test-slave-65tz"
server_key: "68aeee52b2302620c026533b56f81b93"
server_key_iv: "70f450a18957c423a51ad7ee470d9ca6"
server_key_version: "encryptionkey@0"
I20260620 14:49:00.245564 19299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "ac2abe08d40e4147a3ae73ee88101d46"
format_stamp: "Formatted at 2026-06-20 14:49:00 on dist-test-slave-65tz"
server_key: "68aeee52b2302620c026533b56f81b93"
server_key_iv: "70f450a18957c423a51ad7ee470d9ca6"
server_key_version: "encryptionkey@0"
I20260620 14:49:00.249689 19299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:49:00.252594 19314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:00.254393 19299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:49:00.254559 19299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "ac2abe08d40e4147a3ae73ee88101d46"
format_stamp: "Formatted at 2026-06-20 14:49:00 on dist-test-slave-65tz"
server_key: "68aeee52b2302620c026533b56f81b93"
server_key_iv: "70f450a18957c423a51ad7ee470d9ca6"
server_key_version: "encryptionkey@0"
I20260620 14:49:00.254683 19299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:00.285974 19299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:00.289557 19299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:00.289819 19299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:00.298260 19299 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:38881
I20260620 14:49:00.298345 19366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:38881 every 8 connection(s)
I20260620 14:49:00.299438 19299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:49:00.303087 19367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:49:00.308758 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 19299
I20260620 14:49:00.308873 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:49:00.309207 9079 external_mini_cluster.cc:1428] Setting key 4284c478981a0c0aea0c79117cd231b9
I20260620 14:49:00.310247 19367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46: Bootstrap starting.
I20260620 14:49:00.313036 19367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46: Neither blocks nor log segments found. Creating new log.
I20260620 14:49:00.314157 19367 log.cc:824] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:00.316723 19367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46: No bootstrap required, opened a new log
Jun 20 14:49:00 dist-test-slave-65tz krb5kdc[19286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:49:00.321900 19367 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2abe08d40e4147a3ae73ee88101d46" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38881 } }
I20260620 14:49:00.322194 19367 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:49:00.322266 19367 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac2abe08d40e4147a3ae73ee88101d46, State: Initialized, Role: FOLLOWER
I20260620 14:49:00.322798 19367 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2abe08d40e4147a3ae73ee88101d46" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38881 } }
I20260620 14:49:00.322935 19367 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:49:00.323032 19367 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:49:00.323140 19367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:49:00.324536 19367 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2abe08d40e4147a3ae73ee88101d46" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38881 } }
I20260620 14:49:00.325001 19367 leader_election.cc:302] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac2abe08d40e4147a3ae73ee88101d46; no voters:
I20260620 14:49:00.325577 19367 leader_election.cc:288] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:49:00.325752 19372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:49:00.326078 19372 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46 [term 1 LEADER]: Becoming Leader. State: Replica: ac2abe08d40e4147a3ae73ee88101d46, State: Running, Role: LEADER
I20260620 14:49:00.326493 19372 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2abe08d40e4147a3ae73ee88101d46" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38881 } }
I20260620 14:49:00.327385 19367 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:49:00.328320 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:00.311121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36000 (local address 127.8.221.254:38881)
0620 14:49:00.311622 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:00.311639 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:00.311676 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:49:00.312656 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:00.312680 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:00.313084 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:00.313497 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:00.314690 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:00.315797 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:00.317024 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:00.317568 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:00.317712 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:00.320962 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:00.320968 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:00.320985 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:00.321022 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:00.323486 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:00.324101 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:00.324107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:00.324113 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:00.324269 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:00.324722 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:00.324727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:00.324729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:00.325310 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:00.325474 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:00.326662 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0620 14:49:00.327044 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":170,"threads_started":1}
I20260620 14:49:00.328341 19374 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac2abe08d40e4147a3ae73ee88101d46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2abe08d40e4147a3ae73ee88101d46" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38881 } } }
I20260620 14:49:00.328542 19374 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46 [sys.catalog]: This master's current role is: LEADER
I20260620 14:49:00.328881 19373 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac2abe08d40e4147a3ae73ee88101d46. Latest consensus state: current_term: 1 leader_uuid: "ac2abe08d40e4147a3ae73ee88101d46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2abe08d40e4147a3ae73ee88101d46" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38881 } } }
I20260620 14:49:00.328967 19373 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46 [sys.catalog]: This master's current role is: LEADER
I20260620 14:49:00.329246 19381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:49:00.332707 19381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:49:00.339386 19381 catalog_manager.cc:1347] Generated new cluster ID: 93153353767743cc952000fd4db4d3cd
I20260620 14:49:00.339462 19381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:49:00.367051 19381 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:49:00.368497 19381 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:49:00.378127 19381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46: Generated new TSK 0
I20260620 14:49:00.378998 19381 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
I20260620 14:49:00.413115 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38881
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:49:00.538949 19393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:00.539261 19393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:00.539345 19393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:00.543922 19393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:00.544004 19393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:00.544108 19393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:49:00.549695 19393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:49:00.551010 19393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:00.552235 19393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:00.563169 19401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:00.563233 19399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:00.563457 19393 server_base.cc:1034] running on GCE node
W20260620 14:49:00.563547 19398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:00.563987 19393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:00.564716 19393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:00.565876 19393 hybrid_clock.cc:648] HybridClock initialized: now 1781966940565830 us; error 61 us; skew 500 ppm
Jun 20 14:49:00 dist-test-slave-65tz krb5kdc[19286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966940, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:00.568792 19393 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:49:00.569767 19393 webserver.cc:460] Webserver started at http://127.8.221.193:36447/ using document root <none> and password file <none>
I20260620 14:49:00.570473 19393 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:00.570542 19393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:00.570729 19393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:49:00.572906 19393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "152417e0b0bd414483e32aff17981e96"
format_stamp: "Formatted at 2026-06-20 14:49:00 on dist-test-slave-65tz"
server_key: "82132daf97637a3cdf797ae2a8ad4044"
server_key_iv: "61986a030fa1258be9a3a387d1546c59"
server_key_version: "encryptionkey@0"
I20260620 14:49:00.573441 19393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "152417e0b0bd414483e32aff17981e96"
format_stamp: "Formatted at 2026-06-20 14:49:00 on dist-test-slave-65tz"
server_key: "82132daf97637a3cdf797ae2a8ad4044"
server_key_iv: "61986a030fa1258be9a3a387d1546c59"
server_key_version: "encryptionkey@0"
I20260620 14:49:00.577524 19393 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:49:00.580829 19408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:00.582744 19393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:49:00.582898 19393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "152417e0b0bd414483e32aff17981e96"
format_stamp: "Formatted at 2026-06-20 14:49:00 on dist-test-slave-65tz"
server_key: "82132daf97637a3cdf797ae2a8ad4044"
server_key_iv: "61986a030fa1258be9a3a387d1546c59"
server_key_version: "encryptionkey@0"
I20260620 14:49:00.583029 19393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:00.596884 19393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:00.599848 19393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:00.600004 19393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:00.600602 19393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:49:00.601469 19393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:49:00.601545 19393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:00.601616 19393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:49:00.601651 19393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:00.611907 19393 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:41171
I20260620 14:49:00.611987 19521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:41171 every 8 connection(s)
I20260620 14:49:00.612887 19393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:49:00.621223 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 19393
I20260620 14:49:00.621356 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:49:00.621657 9079 external_mini_cluster.cc:1428] Setting key a8390785bd495016f55350c882876a6e
Jun 20 14:49:00 dist-test-slave-65tz krb5kdc[19286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966940, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:49:00.629873 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:00.615351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:42771 (local address 127.8.221.254:38881)
0620 14:49:00.615487 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:00.615495 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:00.616643 (+ 1148us) server_negotiation.cc:407] Connection header received
0620 14:49:00.618224 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:00.618227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:00.618292 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:00.618393 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:00.619606 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:00.620239 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:00.621640 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:00.622030 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:00.622175 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:00.625434 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:00.625437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:00.625440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:00.625467 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:00.627044 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:00.627876 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:00.627879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:00.627880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:00.627955 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:00.628427 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:00.628429 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:00.628431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:00.628617 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:00.628698 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:00.629559 (+ 861us) server_negotiation.cc:299] Negotiation successful
0620 14:49:00.629673 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:49:00.631753 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:00.615808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38881 (local address 127.8.221.193:42771)
0620 14:49:00.616429 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:00.616479 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:00.617984 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:00.618565 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:00.618572 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:00.618961 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:49:00.619455 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:00.619467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:00.620359 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:00.620361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:00.621505 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:00.621517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:00.622223 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:00.622227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:00.622430 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:00.623195 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:49:00.623212 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:49:00.625180 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:49:00.627188 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:00.627198 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:00.627221 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:00.627675 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:00.628073 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:00.628080 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:00.628086 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:00.628311 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:00.628744 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:49:00.628752 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:49:00.629235 (+ 483us) client_negotiation.cc:769] Sending connection context
0620 14:49:00.629534 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 14:49:00.630263 (+ 729us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":192,"threads_started":1}
I20260620 14:49:00.633431 19522 heartbeater.cc:347] Connected to a master server at 127.8.221.254:38881
I20260620 14:49:00.633734 19522 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.221.194@KRBTEST.COM; defaulting to no policy
I20260620 14:49:00.634330 19522 heartbeater.cc:511] Master 127.8.221.254:38881 requested a full tablet report, sending...
I20260620 14:49:00.636143 19331 ts_manager.cc:194] Registered new tserver with Master: 152417e0b0bd414483e32aff17981e96 (127.8.221.193:41171)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.194@KRBTEST.COM" created.
I20260620 14:49:00.637658 19331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:42771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
I20260620 14:49:00.653354 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38881
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:49:00.785745 19527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:00.786193 19527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:00.786298 19527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:00.790663 19527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:00.790724 19527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:00.790827 19527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:49:00.796556 19527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:49:00.797935 19527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:00.799171 19527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:00.810393 19532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:00.810417 19535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:00.810449 19533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:00.810714 19527 server_base.cc:1034] running on GCE node
I20260620 14:49:00.811236 19527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:00.811913 19527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:00.813109 19527 hybrid_clock.cc:648] HybridClock initialized: now 1781966940813089 us; error 38 us; skew 500 ppm
Jun 20 14:49:00 dist-test-slave-65tz krb5kdc[19286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966940, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:00.816399 19527 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:49:00.817406 19527 webserver.cc:460] Webserver started at http://127.8.221.194:44825/ using document root <none> and password file <none>
I20260620 14:49:00.818181 19527 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:00.818274 19527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:00.818555 19527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:49:00.820780 19527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "22ae63902a014048baca138ac8442e7e"
format_stamp: "Formatted at 2026-06-20 14:49:00 on dist-test-slave-65tz"
server_key: "3a0fcf23698750cb837ec59b8bbd5a55"
server_key_iv: "809f4aac7d3d76c1b3c789da2d6958b5"
server_key_version: "encryptionkey@0"
I20260620 14:49:00.821353 19527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "22ae63902a014048baca138ac8442e7e"
format_stamp: "Formatted at 2026-06-20 14:49:00 on dist-test-slave-65tz"
server_key: "3a0fcf23698750cb837ec59b8bbd5a55"
server_key_iv: "809f4aac7d3d76c1b3c789da2d6958b5"
server_key_version: "encryptionkey@0"
I20260620 14:49:00.825423 19527 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 14:49:00.828267 19542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:00.829896 19527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:49:00.830011 19527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "22ae63902a014048baca138ac8442e7e"
format_stamp: "Formatted at 2026-06-20 14:49:00 on dist-test-slave-65tz"
server_key: "3a0fcf23698750cb837ec59b8bbd5a55"
server_key_iv: "809f4aac7d3d76c1b3c789da2d6958b5"
server_key_version: "encryptionkey@0"
I20260620 14:49:00.830101 19527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:00.880803 19527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:00.885182 19527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:00.885464 19527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:00.886270 19527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:49:00.887305 19527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:49:00.887353 19527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:00.887432 19527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:49:00.887467 19527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:00.898983 19527 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:43497
I20260620 14:49:00.899009 19655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:43497 every 8 connection(s)
I20260620 14:49:00.900055 19527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:49:00.901080 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 19527
I20260620 14:49:00.901207 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:49:00.901571 9079 external_mini_cluster.cc:1428] Setting key 1025e50943ad7ae1a954efb1a197707f
Jun 20 14:49:00 dist-test-slave-65tz krb5kdc[19286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966940, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.195@KRBTEST.COM" created.
I20260620 14:49:00.917757 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:00.902389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:51521 (local address 127.8.221.254:38881)
0620 14:49:00.902556 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:00.902561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:00.903556 (+ 995us) server_negotiation.cc:407] Connection header received
0620 14:49:00.905205 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:00.905211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:00.905289 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:00.905388 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:00.907210 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:00.907761 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:00.909754 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:00.910049 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:00.910140 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:00.913610 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:00.913613 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:00.913615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:00.913640 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:00.915327 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:00.915895 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:00.915898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:00.915899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:00.915972 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:00.916335 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:00.916340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:00.916345 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:00.916534 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:00.916659 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:00.917330 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 14:49:00.917473 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:49:00.918972 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:00.902747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38881 (local address 127.8.221.194:51521)
0620 14:49:00.903374 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:00.903440 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:00.904890 (+ 1450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:00.905650 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:00.905663 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:00.906389 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:49:00.907065 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:00.907079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:00.907908 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:00.907914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:00.909609 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:00.909621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:00.910245 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:00.910252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:00.910521 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:00.911292 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:49:00.911322 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:49:00.913392 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:49:00.915462 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:00.915468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:00.915483 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:00.915752 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:00.916068 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:00.916070 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:00.916071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:00.916209 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:00.916691 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:49:00.916702 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:49:00.917050 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 14:49:00.917245 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 14:49:00.917555 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":141,"threads_started":1}
I20260620 14:49:00.920396 19656 heartbeater.cc:347] Connected to a master server at 127.8.221.254:38881
I20260620 14:49:00.920717 19656 heartbeater.cc:464] Registering TS with master...
I20260620 14:49:00.921308 19656 heartbeater.cc:511] Master 127.8.221.254:38881 requested a full tablet report, sending...
I20260620 14:49:00.922600 19331 ts_manager.cc:194] Registered new tserver with Master: 22ae63902a014048baca138ac8442e7e (127.8.221.194:43497)
I20260620 14:49:00.923166 19331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:51521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
I20260620 14:49:00.933298 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38881
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:49:01.060096 19661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:01.060442 19661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:01.060524 19661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:01.064781 19661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:01.064880 19661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:01.064982 19661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:49:01.070473 19661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:49:01.071789 19661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:01.073012 19661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:01.084326 19667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:01.084425 19669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:01.084425 19666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:01.084601 19661 server_base.cc:1034] running on GCE node
I20260620 14:49:01.085139 19661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:01.085924 19661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:01.087216 19661 hybrid_clock.cc:648] HybridClock initialized: now 1781966941087183 us; error 48 us; skew 500 ppm
Jun 20 14:49:01 dist-test-slave-65tz krb5kdc[19286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966941, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:01.090337 19661 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:49:01.091414 19661 webserver.cc:460] Webserver started at http://127.8.221.195:39965/ using document root <none> and password file <none>
I20260620 14:49:01.092079 19661 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:01.092146 19661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:01.092399 19661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:49:01.094868 19661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "20c85e238b7341208b7cab9ec1e0d915"
format_stamp: "Formatted at 2026-06-20 14:49:01 on dist-test-slave-65tz"
server_key: "975de1a6e029e373b73a3cc7c7672d44"
server_key_iv: "86421e92fe1d98d76e6ddb2876844236"
server_key_version: "encryptionkey@0"
I20260620 14:49:01.095551 19661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "20c85e238b7341208b7cab9ec1e0d915"
format_stamp: "Formatted at 2026-06-20 14:49:01 on dist-test-slave-65tz"
server_key: "975de1a6e029e373b73a3cc7c7672d44"
server_key_iv: "86421e92fe1d98d76e6ddb2876844236"
server_key_version: "encryptionkey@0"
I20260620 14:49:01.100092 19661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:49:01.103052 19676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:01.104998 19661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:49:01.105159 19661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "20c85e238b7341208b7cab9ec1e0d915"
format_stamp: "Formatted at 2026-06-20 14:49:01 on dist-test-slave-65tz"
server_key: "975de1a6e029e373b73a3cc7c7672d44"
server_key_iv: "86421e92fe1d98d76e6ddb2876844236"
server_key_version: "encryptionkey@0"
I20260620 14:49:01.105297 19661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:01.117924 19661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:01.121532 19661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:01.121757 19661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:01.122464 19661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:49:01.123451 19661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:49:01.123525 19661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:01.123603 19661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:49:01.123657 19661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:01.134949 19661 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:45431
I20260620 14:49:01.135004 19789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:45431 every 8 connection(s)
I20260620 14:49:01.136068 19661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:49:01.140517 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 19661
I20260620 14:49:01.140697 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:49:01.141021 9079 external_mini_cluster.cc:1428] Setting key bd77cb8cca03c9599d1016eded4d076e
Jun 20 14:49:01 dist-test-slave-65tz krb5kdc[19286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966941, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:49:01.151417 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:01.138329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:48003 (local address 127.8.221.254:38881)
0620 14:49:01.138548 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:01.138554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:01.139639 (+ 1085us) server_negotiation.cc:407] Connection header received
0620 14:49:01.141502 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:01.141506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:01.141567 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:01.141690 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:01.143440 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.143970 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.145117 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.145339 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.145448 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:01.148034 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:01.148037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:01.148039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:01.148060 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:01.149369 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:01.149909 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:01.149913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:01.149914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:01.149994 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:01.150367 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:01.150370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:01.150372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:01.150544 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:01.150651 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:01.151129 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 14:49:01.151247 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:49:01.152225 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:01.138852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38881 (local address 127.8.221.195:48003)
0620 14:49:01.139440 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:01.139497 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:01.141229 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:01.142064 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:01.142072 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:01.142555 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:49:01.143294 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:01.143310 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.144096 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.144099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:01.144989 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:01.144997 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.145493 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.145497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:01.145662 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:01.146330 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:49:01.146346 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:49:01.147862 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:49:01.149485 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:01.149491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:01.149505 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:01.149760 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:01.150107 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:01.150111 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:01.150113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:01.150262 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:01.150650 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:49:01.150656 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:49:01.150891 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 14:49:01.151075 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 14:49:01.151395 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":130,"threads_started":1}
I20260620 14:49:01.153358 19790 heartbeater.cc:347] Connected to a master server at 127.8.221.254:38881
I20260620 14:49:01.153574 19790 heartbeater.cc:464] Registering TS with master...
I20260620 14:49:01.154135 19790 heartbeater.cc:511] Master 127.8.221.254:38881 requested a full tablet report, sending...
I20260620 14:49:01.155418 19330 ts_manager.cc:194] Registered new tserver with Master: 20c85e238b7341208b7cab9ec1e0d915 (127.8.221.195:45431)
I20260620 14:49:01.155970 19330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:48003
I20260620 14:49:01.167577 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:49:01.178062 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:01.169584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36016 (local address 127.8.221.254:38881)
0620 14:49:01.169788 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:01.169820 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:01.170057 (+ 237us) server_negotiation.cc:407] Connection header received
0620 14:49:01.170172 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:01.170175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:01.170231 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:01.170315 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:01.171041 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.171629 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.172439 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.172653 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.172734 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:01.174295 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:01.174298 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:01.174301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:01.174324 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:01.176186 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:01.176595 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:01.176598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:01.176599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:01.176683 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:01.176959 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:01.176961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:01.176963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:01.177232 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:01.177359 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:01.177553 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 14:49:01.177822 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:49:01.181700 19330 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:49:01.185439 19330 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:49:01.202314 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:01.195799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:43497 (local address 127.0.0.1:56038)
0620 14:49:01.196546 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:01.196564 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:01.196663 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:01.197440 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:01.197447 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:01.197470 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:01.197783 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:01.197818 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.199421 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.199425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:01.200612 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:01.200622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.201651 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.201655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:01.201778 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:01.201872 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 14:49:01.201984 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:49:01.202125 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":100,"thread_start_us":137,"threads_started":1}
I20260620 14:49:01.202970 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:01.196397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56038 (local address 127.8.221.194:43497)
0620 14:49:01.196866 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:01.196883 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:01.196912 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:49:01.197013 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:01.197020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:01.197261 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:01.197441 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:01.197974 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.199264 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.200754 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.201483 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.201607 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:01.201772 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:01.202672 (+ 900us) server_negotiation.cc:299] Negotiation successful
0620 14:49:01.202821 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":116,"threads_started":1}
I20260620 14:49:01.203019 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:01.196412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:45431 (local address 127.0.0.1:49672)
0620 14:49:01.196952 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:01.196964 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:01.197051 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:01.197950 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:01.197952 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:01.197976 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:01.198034 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:01.198038 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.199468 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.199494 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:01.201130 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:01.201141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.202514 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.202520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:01.202620 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:01.202649 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 14:49:01.202731 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:49:01.202831 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":136,"thread_start_us":126,"threads_started":1}
I20260620 14:49:01.203290 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:01.195533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:41171 (local address 127.0.0.1:44580)
0620 14:49:01.196274 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:01.196332 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:01.196489 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:01.197545 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:01.197549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:01.197565 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:01.197831 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:01.197836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.199681 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.199685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:01.201130 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:01.201141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.202848 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.202852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:01.202945 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:01.202978 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:49:01.203079 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:49:01.203160 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":224,"thread_start_us":103,"threads_started":1}
I20260620 14:49:01.203321 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:01.196914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49672 (local address 127.8.221.195:45431)
0620 14:49:01.197323 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:01.197333 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:01.197355 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:49:01.197422 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:01.197431 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:01.197603 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:01.197733 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:01.198145 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.199264 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.201289 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.202203 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.202314 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:01.202442 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:01.203032 (+ 590us) server_negotiation.cc:299] Negotiation successful
0620 14:49:01.203155 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":144,"threads_started":1}
I20260620 14:49:01.204475 19802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:01.195687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44580 (local address 127.8.221.193:41171)
0620 14:49:01.197026 (+ 1339us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:01.197036 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:01.197070 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:49:01.197157 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:01.197162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:01.197380 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:01.197683 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:01.198464 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.199547 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.201869 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.202667 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.202812 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:01.202944 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:01.204214 (+ 1270us) server_negotiation.cc:299] Negotiation successful
0620 14:49:01.204319 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1158,"thread_start_us":134,"threads_started":1}
I20260620 14:49:01.206426 19724 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5df266129942a7ba808c6bca404157 (DEFAULT_TABLE table=test-table [id=195d6f7d3a6f4826a807fe3c53c16a7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:49:01.206547 19590 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5df266129942a7ba808c6bca404157 (DEFAULT_TABLE table=test-table [id=195d6f7d3a6f4826a807fe3c53c16a7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:49:01.207154 19456 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5df266129942a7ba808c6bca404157 (DEFAULT_TABLE table=test-table [id=195d6f7d3a6f4826a807fe3c53c16a7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:49:01.207629 19724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5df266129942a7ba808c6bca404157. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:49:01.207813 19590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5df266129942a7ba808c6bca404157. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:49:01.208120 19456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5df266129942a7ba808c6bca404157. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:49:01.215121 19807 tablet_bootstrap.cc:492] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e: Bootstrap starting.
I20260620 14:49:01.215819 19809 tablet_bootstrap.cc:492] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915: Bootstrap starting.
I20260620 14:49:01.216449 19808 tablet_bootstrap.cc:492] T 0d5df266129942a7ba808c6bca404157 P 152417e0b0bd414483e32aff17981e96: Bootstrap starting.
I20260620 14:49:01.218650 19808 tablet_bootstrap.cc:654] T 0d5df266129942a7ba808c6bca404157 P 152417e0b0bd414483e32aff17981e96: Neither blocks nor log segments found. Creating new log.
I20260620 14:49:01.219215 19809 tablet_bootstrap.cc:654] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915: Neither blocks nor log segments found. Creating new log.
I20260620 14:49:01.219756 19808 log.cc:824] T 0d5df266129942a7ba808c6bca404157 P 152417e0b0bd414483e32aff17981e96: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:01.220010 19809 log.cc:824] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:01.220764 19807 tablet_bootstrap.cc:654] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e: Neither blocks nor log segments found. Creating new log.
I20260620 14:49:01.221520 19807 log.cc:824] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:01.222167 19808 tablet_bootstrap.cc:492] T 0d5df266129942a7ba808c6bca404157 P 152417e0b0bd414483e32aff17981e96: No bootstrap required, opened a new log
I20260620 14:49:01.222395 19808 ts_tablet_manager.cc:1397] T 0d5df266129942a7ba808c6bca404157 P 152417e0b0bd414483e32aff17981e96: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:49:01.223017 19809 tablet_bootstrap.cc:492] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915: No bootstrap required, opened a new log
I20260620 14:49:01.223181 19809 ts_tablet_manager.cc:1397] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 14:49:01.223656 19807 tablet_bootstrap.cc:492] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e: No bootstrap required, opened a new log
I20260620 14:49:01.223865 19807 ts_tablet_manager.cc:1397] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260620 14:49:01.226893 19807 raft_consensus.cc:348] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152417e0b0bd414483e32aff17981e96" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41171 } } peers { permanent_uuid: "20c85e238b7341208b7cab9ec1e0d915" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45431 } } peers { permanent_uuid: "22ae63902a014048baca138ac8442e7e" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43497 } }
I20260620 14:49:01.227141 19807 raft_consensus.cc:374] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:49:01.227203 19807 raft_consensus.cc:729] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ae63902a014048baca138ac8442e7e, State: Initialized, Role: FOLLOWER
I20260620 14:49:01.227355 19809 raft_consensus.cc:348] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152417e0b0bd414483e32aff17981e96" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41171 } } peers { permanent_uuid: "20c85e238b7341208b7cab9ec1e0d915" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45431 } } peers { permanent_uuid: "22ae63902a014048baca138ac8442e7e" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43497 } }
I20260620 14:49:01.227424 19808 raft_consensus.cc:348] T 0d5df266129942a7ba808c6bca404157 P 152417e0b0bd414483e32aff17981e96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152417e0b0bd414483e32aff17981e96" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41171 } } peers { permanent_uuid: "20c85e238b7341208b7cab9ec1e0d915" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45431 } } peers { permanent_uuid: "22ae63902a014048baca138ac8442e7e" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43497 } }
I20260620 14:49:01.227645 19809 raft_consensus.cc:374] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:49:01.227645 19808 raft_consensus.cc:374] T 0d5df266129942a7ba808c6bca404157 P 152417e0b0bd414483e32aff17981e96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:49:01.227705 19809 raft_consensus.cc:729] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c85e238b7341208b7cab9ec1e0d915, State: Initialized, Role: FOLLOWER
I20260620 14:49:01.227713 19808 raft_consensus.cc:729] T 0d5df266129942a7ba808c6bca404157 P 152417e0b0bd414483e32aff17981e96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152417e0b0bd414483e32aff17981e96, State: Initialized, Role: FOLLOWER
I20260620 14:49:01.227728 19807 consensus_queue.cc:260] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152417e0b0bd414483e32aff17981e96" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41171 } } peers { permanent_uuid: "20c85e238b7341208b7cab9ec1e0d915" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45431 } } peers { permanent_uuid: "22ae63902a014048baca138ac8442e7e" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43497 } }
I20260620 14:49:01.228214 19808 consensus_queue.cc:260] T 0d5df266129942a7ba808c6bca404157 P 152417e0b0bd414483e32aff17981e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152417e0b0bd414483e32aff17981e96" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41171 } } peers { permanent_uuid: "20c85e238b7341208b7cab9ec1e0d915" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45431 } } peers { permanent_uuid: "22ae63902a014048baca138ac8442e7e" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43497 } }
I20260620 14:49:01.228214 19809 consensus_queue.cc:260] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152417e0b0bd414483e32aff17981e96" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41171 } } peers { permanent_uuid: "20c85e238b7341208b7cab9ec1e0d915" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45431 } } peers { permanent_uuid: "22ae63902a014048baca138ac8442e7e" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43497 } }
I20260620 14:49:01.228417 19656 heartbeater.cc:503] Master 127.8.221.254:38881 was elected leader, sending a full tablet report...
I20260620 14:49:01.228834 19807 ts_tablet_manager.cc:1428] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:49:01.229221 19808 ts_tablet_manager.cc:1428] T 0d5df266129942a7ba808c6bca404157 P 152417e0b0bd414483e32aff17981e96: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 14:49:01.229221 19809 ts_tablet_manager.cc:1428] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:49:01.229576 19522 heartbeater.cc:503] Master 127.8.221.254:38881 was elected leader, sending a full tablet report...
I20260620 14:49:01.229576 19790 heartbeater.cc:503] Master 127.8.221.254:38881 was elected leader, sending a full tablet report...
I20260620 14:49:01.294323 19813 raft_consensus.cc:482] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:49:01.294538 19813 raft_consensus.cc:504] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152417e0b0bd414483e32aff17981e96" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41171 } } peers { permanent_uuid: "20c85e238b7341208b7cab9ec1e0d915" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45431 } } peers { permanent_uuid: "22ae63902a014048baca138ac8442e7e" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43497 } }
I20260620 14:49:01.295670 19813 leader_election.cc:288] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 152417e0b0bd414483e32aff17981e96 (127.8.221.193:41171), 20c85e238b7341208b7cab9ec1e0d915 (127.8.221.195:45431)
I20260620 14:49:01.300791 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:01.296583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:33247 (local address 127.8.221.195:45431)
0620 14:49:01.296717 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:01.296721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:01.296940 (+ 219us) server_negotiation.cc:407] Connection header received
0620 14:49:01.297064 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:01.297068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:01.297122 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:01.297187 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:01.297546 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.298245 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.299683 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.300133 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.300277 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:01.300344 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:01.300554 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 14:49:01.300657 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:49:01.300777 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:01.296067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:41171 (local address 127.8.221.194:46735)
0620 14:49:01.296245 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:01.296265 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:01.296382 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:01.296822 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:01.296825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:01.296840 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:01.296974 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:01.296983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.298111 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.298114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:01.299409 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:01.299419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.300424 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.300427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:01.300548 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:01.300565 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:49:01.300629 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:49:01.300695 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 14:49:01.300786 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:01.296383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:45431 (local address 127.8.221.194:33247)
0620 14:49:01.296822 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:01.296836 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:01.296954 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:01.297222 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:01.297225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:01.297242 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:01.297420 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:01.297428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.298373 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.298377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:01.299531 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:01.299541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.300273 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.300277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:01.300394 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:01.300426 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 14:49:01.300515 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:49:01.300602 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":197,"threads_started":1}
I20260620 14:49:01.301349 19802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:01.296212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:46735 (local address 127.8.221.193:41171)
0620 14:49:01.296359 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:01.296363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:01.296378 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:49:01.296560 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:01.296563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:01.296613 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:01.296693 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:01.297136 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.297967 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.299574 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.300236 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.300359 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:01.300413 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:01.301128 (+ 715us) server_negotiation.cc:299] Negotiation successful
0620 14:49:01.301245 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:49:01.302095 19476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5df266129942a7ba808c6bca404157" candidate_uuid: "22ae63902a014048baca138ac8442e7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152417e0b0bd414483e32aff17981e96" is_pre_election: true
I20260620 14:49:01.302141 19744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5df266129942a7ba808c6bca404157" candidate_uuid: "22ae63902a014048baca138ac8442e7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20c85e238b7341208b7cab9ec1e0d915" is_pre_election: true
I20260620 14:49:01.302435 19744 raft_consensus.cc:2454] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ae63902a014048baca138ac8442e7e in term 0.
I20260620 14:49:01.302489 19476 raft_consensus.cc:2454] T 0d5df266129942a7ba808c6bca404157 P 152417e0b0bd414483e32aff17981e96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22ae63902a014048baca138ac8442e7e in term 0.
I20260620 14:49:01.302943 19546 leader_election.cc:302] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20c85e238b7341208b7cab9ec1e0d915, 22ae63902a014048baca138ac8442e7e; no voters:
I20260620 14:49:01.303249 19813 raft_consensus.cc:2781] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:49:01.303356 19813 raft_consensus.cc:482] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:49:01.303414 19813 raft_consensus.cc:3037] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:49:01.304664 19813 raft_consensus.cc:504] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152417e0b0bd414483e32aff17981e96" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41171 } } peers { permanent_uuid: "20c85e238b7341208b7cab9ec1e0d915" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45431 } } peers { permanent_uuid: "22ae63902a014048baca138ac8442e7e" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43497 } }
I20260620 14:49:01.305070 19813 leader_election.cc:288] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [CANDIDATE]: Term 1 election: Requested vote from peers 152417e0b0bd414483e32aff17981e96 (127.8.221.193:41171), 20c85e238b7341208b7cab9ec1e0d915 (127.8.221.195:45431)
I20260620 14:49:01.305552 19744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5df266129942a7ba808c6bca404157" candidate_uuid: "22ae63902a014048baca138ac8442e7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20c85e238b7341208b7cab9ec1e0d915"
I20260620 14:49:01.305536 19476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5df266129942a7ba808c6bca404157" candidate_uuid: "22ae63902a014048baca138ac8442e7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152417e0b0bd414483e32aff17981e96"
I20260620 14:49:01.305711 19744 raft_consensus.cc:3037] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:49:01.305713 19476 raft_consensus.cc:3037] T 0d5df266129942a7ba808c6bca404157 P 152417e0b0bd414483e32aff17981e96 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:49:01.306926 19744 raft_consensus.cc:2454] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ae63902a014048baca138ac8442e7e in term 1.
I20260620 14:49:01.307143 19476 raft_consensus.cc:2454] T 0d5df266129942a7ba808c6bca404157 P 152417e0b0bd414483e32aff17981e96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22ae63902a014048baca138ac8442e7e in term 1.
I20260620 14:49:01.307233 19546 leader_election.cc:302] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20c85e238b7341208b7cab9ec1e0d915, 22ae63902a014048baca138ac8442e7e; no voters:
I20260620 14:49:01.307520 19813 raft_consensus.cc:2781] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:49:01.307787 19813 raft_consensus.cc:686] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [term 1 LEADER]: Becoming Leader. State: Replica: 22ae63902a014048baca138ac8442e7e, State: Running, Role: LEADER
I20260620 14:49:01.308233 19813 consensus_queue.cc:237] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152417e0b0bd414483e32aff17981e96" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41171 } } peers { permanent_uuid: "20c85e238b7341208b7cab9ec1e0d915" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45431 } } peers { permanent_uuid: "22ae63902a014048baca138ac8442e7e" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43497 } }
I20260620 14:49:01.312178 19330 catalog_manager.cc:5495] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 22ae63902a014048baca138ac8442e7e (127.8.221.194). New cstate: current_term: 1 leader_uuid: "22ae63902a014048baca138ac8442e7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152417e0b0bd414483e32aff17981e96" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20c85e238b7341208b7cab9ec1e0d915" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ae63902a014048baca138ac8442e7e" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 43497 } health_report { overall_health: HEALTHY } } }
I20260620 14:49:01.323235 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:01.319773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56048 (local address 127.8.221.194:43497)
0620 14:49:01.319885 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:01.319888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:01.319919 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:49:01.320063 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:01.320065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:01.320117 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:01.320217 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:49:01.320457 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.320945 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.321849 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:01.322061 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:01.322142 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:01.322403 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:49:01.322780 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:49:01.322871 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:01.322959 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 14:49:01.323015 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:49:01.329473 19476 raft_consensus.cc:1261] T 0d5df266129942a7ba808c6bca404157 P 152417e0b0bd414483e32aff17981e96 [term 1 FOLLOWER]: Refusing update from remote peer 22ae63902a014048baca138ac8442e7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:49:01.329479 19744 raft_consensus.cc:1261] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915 [term 1 FOLLOWER]: Refusing update from remote peer 22ae63902a014048baca138ac8442e7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:49:01.330216 19813 consensus_queue.cc:1035] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "152417e0b0bd414483e32aff17981e96" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 41171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:49:01.330439 19817 consensus_queue.cc:1035] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "20c85e238b7341208b7cab9ec1e0d915" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:49:01.340977 19822 mvcc.cc:205] Tried to move back new op lower bound from 7298936591677988864 to 7298936591602266112. Current Snapshot: MvccSnapshot[applied={T|T < 7298936591677988864}]
I20260620 14:49:01.349911 19330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:49:01.350194 19330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:49:01.353052 19330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac2abe08d40e4147a3ae73ee88101d46: Sending DeleteTablet for 3 replicas of tablet 0d5df266129942a7ba808c6bca404157
I20260620 14:49:01.354096 19456 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5df266129942a7ba808c6bca404157 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:49:01 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:44580
I20260620 14:49:01.354103 19590 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5df266129942a7ba808c6bca404157 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:49:01 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:56038
I20260620 14:49:01.354316 19724 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5df266129942a7ba808c6bca404157 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:49:01 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:49672
I20260620 14:49:01.354679 19830 tablet_replica.cc:330] T 0d5df266129942a7ba808c6bca404157 P 152417e0b0bd414483e32aff17981e96: stopping tablet replica
I20260620 14:49:01.355033 19830 raft_consensus.cc:2229] T 0d5df266129942a7ba808c6bca404157 P 152417e0b0bd414483e32aff17981e96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:49:01.355180 19832 tablet_replica.cc:330] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915: stopping tablet replica
I20260620 14:49:01.355402 19830 raft_consensus.cc:2258] T 0d5df266129942a7ba808c6bca404157 P 152417e0b0bd414483e32aff17981e96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:49:01.355583 19832 raft_consensus.cc:2229] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:49:01.355836 19831 tablet_replica.cc:330] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e: stopping tablet replica
I20260620 14:49:01.355911 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 19393
I20260620 14:49:01.356060 19832 raft_consensus.cc:2258] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:49:01.356117 19831 raft_consensus.cc:2229] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:49:01.356568 19831 raft_consensus.cc:2258] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:49:01.359653 19832 ts_tablet_manager.cc:1905] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:49:01.359899 19831 ts_tablet_manager.cc:1905] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:49:01.362877 19831 ts_tablet_manager.cc:1918] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:49:01.362993 19831 log.cc:1196] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/0d5df266129942a7ba808c6bca404157
I20260620 14:49:01.363317 19831 ts_tablet_manager.cc:1939] T 0d5df266129942a7ba808c6bca404157 P 22ae63902a014048baca138ac8442e7e: Deleting consensus metadata
I20260620 14:49:01.363858 19832 ts_tablet_manager.cc:1918] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:49:01.364024 19832 log.cc:1196] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/0d5df266129942a7ba808c6bca404157
I20260620 14:49:01.364419 19832 ts_tablet_manager.cc:1939] T 0d5df266129942a7ba808c6bca404157 P 20c85e238b7341208b7cab9ec1e0d915: Deleting consensus metadata
I20260620 14:49:01.364674 19316 catalog_manager.cc:4841] TS 22ae63902a014048baca138ac8442e7e (127.8.221.194:43497): tablet 0d5df266129942a7ba808c6bca404157 (table test-table [id=195d6f7d3a6f4826a807fe3c53c16a7c]) successfully deleted
W20260620 14:49:01.365840 19318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.221.193:41171 (error 108)
I20260620 14:49:01.366452 19318 catalog_manager.cc:4841] TS 20c85e238b7341208b7cab9ec1e0d915 (127.8.221.195:45431): tablet 0d5df266129942a7ba808c6bca404157 (table test-table [id=195d6f7d3a6f4826a807fe3c53c16a7c]) successfully deleted
W20260620 14:49:01.367131 19803 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:49:01.366777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:41171 (local address 127.0.0.1:44592)
0620 14:49:01.366936 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:01.367027 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.193:41171: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 14:49:01.367295 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 19527
W20260620 14:49:01.367370 19318 catalog_manager.cc:4584] TS 152417e0b0bd414483e32aff17981e96 (127.8.221.193:41171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d5df266129942a7ba808c6bca404157: Network error: Client connection negotiation failed: client connection to 127.8.221.193:41171: connect: Connection reset by peer (error 104)
I20260620 14:49:01.375176 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 19661
I20260620 14:49:01.382113 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 19299
2026-06-20T14:49:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2510 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:49:01 dist-test-slave-65tz krb5kdc[19836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:49:01 dist-test-slave-65tz krb5kdc[19836](info): set up 2 sockets
Jun 20 14:49:01 dist-test-slave-65tz krb5kdc[19836](info): commencing operation
krb5kdc: starting...
W20260620 14:49:02.447295 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:49:02 dist-test-slave-65tz krb5kdc[19836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:49:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:49:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.221.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.221.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.221.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/rangeradmin_127.8.221.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.221.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/rangeradmin_127.8.221.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.221.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.221.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.221.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/rangerlookup_127.8.221.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.221.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/rangerlookup_127.8.221.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.221.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.221.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.221.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/HTTP_127.8.221.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.221.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/HTTP_127.8.221.212@KRBTEST.COM.keytab.
I20260620 14:49:02.634864 9079 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQhiojW/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/postgres -l logfile start
2026-06-20 14:49:05.232 UTC [19866] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:49:05.232 UTC [19866] LOG: listening on IPv4 address "127.8.221.212", port 56345
2026-06-20 14:49:05.235 UTC [19866] LOG: listening on Unix socket "/tmp/.s.PGSQL.56345"
2026-06-20 14:49:05.258 UTC [19869] LOG: database system was shut down at 2026-06-20 14:49:03 UTC
2026-06-20 14:49:05.262 UTC [19866] LOG: database system is ready to accept connections
I20260620 14:49:06.215093 9079 mini_postgres.cc:96] Postgres bound to 56345
2026-06-20 14:49:06.221 UTC [19877] FATAL: database "slave" does not exist
127.8.221.212:56345 - accepting connections
I20260620 14:49:06.222729 9079 mini_ranger.cc:161] Starting Ranger...
I20260620 14:49:06.238972 9079 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:49:06.729889 9079 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:49:06.729993 9079 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ranger-admin
2026-06-20 14:49:07,002 [I] DB FLAVOR :POSTGRES
2026-06-20 14:49:07,002 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:49:07,003 [I] Checking connection..
2026-06-20 14:49:07,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:49:07,279 [I] Checking connection passed.
2026-06-20 14:49:07,280 [I] --------- Verifying version history table ---------
2026-06-20 14:49:07,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:49:07,554 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:49:07,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:49:07,809 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:49:07,809 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:49:07,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:49:08.068 UTC [19941] WARNING: there is no transaction in progress
2026-06-20 14:49:08,075 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:49:08,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:49:08,356 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:49:08,356 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:49:08,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:49:08,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:49:08,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-65tz.c.gcp-upstream.internal','N') ;"
2026-06-20 14:49:09,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:49:09,423 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:49:09,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:49:09,663 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:49:09,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:49:09,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:49:10,212 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:49:10,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:49:11.285 UTC [20049] WARNING: there is no transaction in progress
2026-06-20 14:49:11.303 UTC [20049] WARNING: there is no transaction in progress
2026-06-20 14:49:11.320 UTC [20049] WARNING: there is no transaction in progress
2026-06-20 14:49:11.337 UTC [20049] WARNING: there is no transaction in progress
2026-06-20 14:49:11.354 UTC [20049] WARNING: there is no transaction in progress
2026-06-20 14:49:11.374 UTC [20049] WARNING: there is no transaction in progress
2026-06-20 14:49:11.437 UTC [20049] WARNING: there is no transaction in progress
2026-06-20 14:49:11.445 UTC [20049] WARNING: there is no transaction in progress
2026-06-20 14:49:11.461 UTC [20049] WARNING: there is no transaction in progress
2026-06-20 14:49:11.472 UTC [20049] WARNING: there is no transaction in progress
2026-06-20 14:49:11.484 UTC [20049] WARNING: there is no transaction in progress
2026-06-20 14:49:11.499 UTC [20049] WARNING: there is no transaction in progress
2026-06-20 14:49:11.510 UTC [20049] WARNING: there is no transaction in progress
2026-06-20 14:49:11.517 UTC [20049] WARNING: there is no transaction in progress
2026-06-20 14:49:11.524 UTC [20049] WARNING: there is no transaction in progress
2026-06-20 14:49:11.532 UTC [20049] WARNING: there is no transaction in progress
2026-06-20 14:49:12,281 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:49:12,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:49:12,540 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:49:12,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:49:12,833 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:49:12,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:49:13,105 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:49:13,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:49:13,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:49:13,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-65tz.c.gcp-upstream.internal';"
2026-06-20 14:49:13,894 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:49:13,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:49:14,156 [I] DB_PATCHES have already been applied
I20260620 14:49:14.161993 9079 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskQhiojW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:49:14.162084 9079 mini_ranger.cc:193] Using host: 127.8.221.212
I20260620 14:49:14.166723 9079 mini_ranger.cc:227] Ranger admin URL: http://127.8.221.212:35137
Jun 20, 2026 2:49:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:49:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:49:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:49:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:49:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:49:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:49:15 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:49:15.297704 9079 mini_ranger.cc:160] Time spent starting Ranger: real 9.075s user 0.006s sys 0.006s
Jun 20, 2026 2:49:36 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:49:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 2:49:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:49:36 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:49:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:49:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:49:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:49:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:49:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:49:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:49:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:49:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:49:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:49:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:49:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:49:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:49:36 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:49:37 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:49:38.314450 9079 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.221.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.221.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.221.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/rangerkms_127.8.221.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.221.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/rangerkms_127.8.221.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.221.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.221.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.221.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/HTTP_127.8.221.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.221.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/HTTP_127.8.221.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:49:38 dist-test-slave-65tz krb5kdc[19836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966978, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:49:38.413314 9079 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:49:38.425086 9079 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:49:39.350142 9079 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:49:39.350229 9079 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ranger-kms
I20260620 14:49:39.350355 9079 mini_ranger_kms.cc:227] Using postgres at 127.8.221.212:56345
2026-06-20 14:49:39,540 [I] DB FLAVOR :POSTGRES
2026-06-20 14:49:39,541 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:49:39,541 [I] Checking connection
2026-06-20 14:49:39,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:49:39,837 [I] connection success
2026-06-20 14:49:39,838 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:49:39,838 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:49:39,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:49:40,112 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:49:40,113 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:49:40,113 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:49:40,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:56345/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:49:40,407 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:49:40.939522 9079 mini_ranger_kms.cc:310] Created kms service
I20260620 14:49:41.191244 9079 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:49:41.235363 9079 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:49:41 dist-test-slave-65tz krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966978, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
I20260620 14:49:41.634454 9079 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:49:41.634660 9079 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:49:41.634708 9079 mini_ranger_kms.cc:240] Using host: 127.8.221.212
I20260620 14:49:41.640216 9079 mini_ranger_kms.cc:276] Ranger KMS PID: 20230
I20260620 14:49:41.640424 9079 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.221.212:52571
Jun 20, 2026 2:49:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:49:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:49:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:49:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:49:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:49:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 14:49:44.038552 9079 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.625s user 0.004s sys 0.168s
I20260620 14:49:44.038710 9079 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:49:44.038836 9079 mini_ranger_kms.cc:401] 127.8.221.212:52571/kms/v1/keys
Jun 20 14:49:44 dist-test-slave-65tz krb5kdc[19836](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781966984, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.221.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 14:49:44 dist-test-slave-65tz krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966984, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.221.212@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
Jun 20 14:49:45 dist-test-slave-65tz krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966984, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.221.212@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
Jun 20 14:49:47 dist-test-slave-65tz krb5kdc[19836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:49:47.713367 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:42163
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:46333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:42163
--ranger_config_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.221.212:52571/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:49:47.866767 20280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:47.867065 20280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:47.867107 20280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:47.871315 20280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:49:47.871390 20280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:47.871410 20280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:47.871447 20280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:49:47.871510 20280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:49:47.877293 20280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:46333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.221.212:52571/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:42163
--ranger_config_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:42163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:49:47.878684 20280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:47.879972 20280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:47.890414 20288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:47.890580 20285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:47.890414 20286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:47.890718 20280 server_base.cc:1034] running on GCE node
I20260620 14:49:47.891635 20280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:47.892987 20280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:47.894179 20280 hybrid_clock.cc:648] HybridClock initialized: now 1781966987894148 us; error 54 us; skew 500 ppm
Jun 20 14:49:47 dist-test-slave-65tz krb5kdc[19836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966987, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:47.906208 20280 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:49:47.907657 20280 webserver.cc:460] Webserver started at http://127.8.221.254:36573/ using document root <none> and password file <none>
I20260620 14:49:47.908390 20280 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:47.908480 20280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:47.908768 20280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:49:47 dist-test-slave-65tz krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966987, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
I20260620 14:49:47.970522 20280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "4889d5ee0a0846e7a5ea3fa237390d5c"
format_stamp: "Formatted at 2026-06-20 14:49:47 on dist-test-slave-65tz"
server_key: "a89be37b9094522ed2f3039bebc3fcfa"
server_key_iv: "bed703d434c9ebed705f5ce78dac189c"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:47.971284 20280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "4889d5ee0a0846e7a5ea3fa237390d5c"
format_stamp: "Formatted at 2026-06-20 14:49:47 on dist-test-slave-65tz"
server_key: "a89be37b9094522ed2f3039bebc3fcfa"
server_key_iv: "bed703d434c9ebed705f5ce78dac189c"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:47.976677 20280 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:49:47.996999 20295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:47.999126 20280 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 14:49:47.999323 20280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "4889d5ee0a0846e7a5ea3fa237390d5c"
format_stamp: "Formatted at 2026-06-20 14:49:47 on dist-test-slave-65tz"
server_key: "a89be37b9094522ed2f3039bebc3fcfa"
server_key_iv: "bed703d434c9ebed705f5ce78dac189c"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:47.999491 20280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:48.020910 20280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:48.024425 20280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:48.024638 20280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:48.033959 20280 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:42163
I20260620 14:49:48.034021 20347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:42163 every 8 connection(s)
I20260620 14:49:48.035270 20280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:49:48.039000 20348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:49:48.044342 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 20280
I20260620 14:49:48.044502 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:49:48.044852 9079 external_mini_cluster.cc:1428] Setting key 82b1c951babe7804f8d929b1c1e9d6d0
I20260620 14:49:48.046507 20348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c: Bootstrap starting.
I20260620 14:49:48.049422 20348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c: Neither blocks nor log segments found. Creating new log.
I20260620 14:49:48.050736 20348 log.cc:824] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:48.053084 20348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c: No bootstrap required, opened a new log
Jun 20 14:49:48 dist-test-slave-65tz krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:49:48.058085 20348 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4889d5ee0a0846e7a5ea3fa237390d5c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 42163 } }
I20260620 14:49:48.058450 20348 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:49:48.058533 20348 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4889d5ee0a0846e7a5ea3fa237390d5c, State: Initialized, Role: FOLLOWER
I20260620 14:49:48.059149 20348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4889d5ee0a0846e7a5ea3fa237390d5c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 42163 } }
I20260620 14:49:48.059365 20348 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:49:48.059473 20348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:49:48.059618 20348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:49:48.060930 20348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4889d5ee0a0846e7a5ea3fa237390d5c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 42163 } }
I20260620 14:49:48.061379 20348 leader_election.cc:302] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4889d5ee0a0846e7a5ea3fa237390d5c; no voters:
I20260620 14:49:48.061770 20348 leader_election.cc:288] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:49:48.061960 20353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:49:48.063331 20348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:49:48.063838 20351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:48.046782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54016 (local address 127.8.221.254:42163)
0620 14:49:48.047477 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:48.047501 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:48.047561 (+ 60us) server_negotiation.cc:407] Connection header received
0620 14:49:48.048620 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:48.048647 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:48.049169 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:48.049570 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:48.050731 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:48.052285 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:48.053552 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:48.054302 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:48.054476 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:48.058008 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:48.058016 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:48.058034 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:48.058070 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:48.060201 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:48.060776 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:48.060784 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:48.060792 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:48.060940 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:48.061334 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:48.061337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:48.061338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:48.061983 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:48.062238 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:48.062509 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 14:49:48.062818 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":230,"threads_started":1}
I20260620 14:49:48.066489 20353 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c [term 1 LEADER]: Becoming Leader. State: Replica: 4889d5ee0a0846e7a5ea3fa237390d5c, State: Running, Role: LEADER
I20260620 14:49:48.066861 20353 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4889d5ee0a0846e7a5ea3fa237390d5c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 42163 } }
I20260620 14:49:48.069276 20365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4889d5ee0a0846e7a5ea3fa237390d5c. Latest consensus state: current_term: 1 leader_uuid: "4889d5ee0a0846e7a5ea3fa237390d5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4889d5ee0a0846e7a5ea3fa237390d5c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 42163 } } }
I20260620 14:49:48.069578 20365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c [sys.catalog]: This master's current role is: LEADER
I20260620 14:49:48.069679 20348 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:49:48.070739 20367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:49:48.070770 20354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4889d5ee0a0846e7a5ea3fa237390d5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4889d5ee0a0846e7a5ea3fa237390d5c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 42163 } } }
I20260620 14:49:48.070930 20354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c [sys.catalog]: This master's current role is: LEADER
I20260620 14:49:48.077373 20367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:49:48.086506 20367 catalog_manager.cc:1347] Generated new cluster ID: 1c26baa46b3041d4a023e1e22d19f4af
I20260620 14:49:48.086584 20367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:49:48.109076 20367 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:49:48.110750 20367 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:49:48.116992 20367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c: Generated new TSK 0
I20260620 14:49:48.118072 20367 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
I20260620 14:49:49.005656 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.221.212:52571/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:42163
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:46333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:49:49.162626 20393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:49.163035 20393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:49.163120 20393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:49.168193 20393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:49.168313 20393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:49.168409 20393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:49:49.177143 20393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:46333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.221.212:52571/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:42163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:49:49.178949 20393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:49.180570 20393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:49.193874 20400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:49.193902 20403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:49.207147 20393 server_base.cc:1034] running on GCE node
W20260620 14:49:49.207468 20401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:49.208173 20393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:49.209105 20393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:49.210330 20393 hybrid_clock.cc:648] HybridClock initialized: now 1781966989210295 us; error 54 us; skew 500 ppm
Jun 20 14:49:49 dist-test-slave-65tz krb5kdc[19836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966989, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:49.214639 20393 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:49:49.215905 20393 webserver.cc:460] Webserver started at http://127.8.221.193:44559/ using document root <none> and password file <none>
I20260620 14:49:49.216717 20393 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:49.216877 20393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:49.217182 20393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:49:49 dist-test-slave-65tz krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966989, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
I20260620 14:49:49.240859 20393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "08e72c0b004c4f488216fd594e434ebc"
format_stamp: "Formatted at 2026-06-20 14:49:49 on dist-test-slave-65tz"
server_key: "e427391bc3323912a22edec065266cbc"
server_key_iv: "e7c807e437f01b3fade9576faea83d4a"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:49.241829 20393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "08e72c0b004c4f488216fd594e434ebc"
format_stamp: "Formatted at 2026-06-20 14:49:49 on dist-test-slave-65tz"
server_key: "e427391bc3323912a22edec065266cbc"
server_key_iv: "e7c807e437f01b3fade9576faea83d4a"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:49.247625 20393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 14:49:49.278173 20412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:49.279907 20393 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.001s
I20260620 14:49:49.280033 20393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "08e72c0b004c4f488216fd594e434ebc"
format_stamp: "Formatted at 2026-06-20 14:49:49 on dist-test-slave-65tz"
server_key: "e427391bc3323912a22edec065266cbc"
server_key_iv: "e7c807e437f01b3fade9576faea83d4a"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:49.280166 20393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:49.306708 20393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:49.310973 20393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:49.311175 20393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:49.311908 20393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:49:49.313158 20393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:49:49.313273 20393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:49.313427 20393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:49:49.313512 20393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:49:49 dist-test-slave-65tz krb5kdc[19836](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781966989, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:49.326802 20393 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:43531
I20260620 14:49:49.327914 20393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:49:49.328610 20525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:43531 every 8 connection(s)
I20260620 14:49:49.329766 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 20393
I20260620 14:49:49.329972 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:49:49.330358 9079 external_mini_cluster.cc:1428] Setting key ce0d1331e91813388804f4ea4f0c4696
Jun 20 14:49:49 dist-test-slave-65tz krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966989, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:49:49.351006 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:49.331334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:42163 (local address 127.8.221.193:45281)
0620 14:49:49.332838 (+ 1504us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:49.332891 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:49.333935 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:49.334895 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:49.334904 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:49.335232 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:49:49.335854 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:49.335871 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:49.337025 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:49.337028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:49.338367 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:49.338378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:49.339145 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:49.339150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:49.339353 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:49.343201 (+ 3848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:49:49.343221 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:49:49.345002 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:49:49.347725 (+ 2723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:49.347730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:49.347746 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:49.348092 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:49.348451 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:49.348455 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:49.348457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:49.348623 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:49.349117 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:49:49.349124 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:49:49.349442 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 14:49:49.349633 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 14:49:49.349975 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1287,"thread_start_us":139,"threads_started":1}
I20260620 14:49:49.351851 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:49.332407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:45281 (local address 127.8.221.254:42163)
0620 14:49:49.334337 (+ 1930us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:49.334342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:49.334356 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:49:49.334402 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:49.334405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:49.334452 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:49.334539 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:49.336015 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:49.336885 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:49.338500 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:49.338851 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:49.338950 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:49.345197 (+ 6247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:49.345200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:49.345203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:49.345229 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:49.347541 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:49.348224 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:49.348228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:49.348230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:49.348349 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:49.348726 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:49.348729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:49.348731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:49.348969 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:49.349065 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:49.351543 (+ 2478us) server_negotiation.cc:299] Negotiation successful
0620 14:49:49.351682 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1843,"thread_start_us":66,"threads_started":1}
I20260620 14:49:49.353237 20526 heartbeater.cc:347] Connected to a master server at 127.8.221.254:42163
I20260620 14:49:49.353561 20526 heartbeater.cc:464] Registering TS with master...
I20260620 14:49:49.354136 20526 heartbeater.cc:511] Master 127.8.221.254:42163 requested a full tablet report, sending...
I20260620 14:49:49.355954 20305 ts_manager.cc:194] Registered new tserver with Master: 08e72c0b004c4f488216fd594e434ebc (127.8.221.193:43531)
I20260620 14:49:49.357475 20305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:45281
WARNING: no policy specified for kudu/127.8.221.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
I20260620 14:49:49.385430 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.221.212:52571/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:42163
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:46333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:49:49.571768 20532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:49.572252 20532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:49.572328 20532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:49.579468 20532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:49.579594 20532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:49.579703 20532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:49:49.586944 20532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:46333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.221.212:52571/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:42163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:49:49.588510 20532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:49.589886 20532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:49.607383 20538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:49.607728 20539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:49.608119 20532 server_base.cc:1034] running on GCE node
W20260620 14:49:49.608412 20541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:49.608897 20532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:49.609880 20532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:49.611121 20532 hybrid_clock.cc:648] HybridClock initialized: now 1781966989611066 us; error 99 us; skew 500 ppm
Jun 20 14:49:49 dist-test-slave-65tz krb5kdc[19836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966989, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:49.615096 20532 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:49:49.616377 20532 webserver.cc:460] Webserver started at http://127.8.221.194:39189/ using document root <none> and password file <none>
I20260620 14:49:49.617280 20532 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:49.617434 20532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:49.617815 20532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:49:49 dist-test-slave-65tz krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966989, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
I20260620 14:49:49.642459 20532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "abffa0d26b284d269ad4d60b9241b6f5"
format_stamp: "Formatted at 2026-06-20 14:49:49 on dist-test-slave-65tz"
server_key: "692e88d91a65fa602f3d4eeea80a088d"
server_key_iv: "0fd6cf72645865ce911832a14c8028a1"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:49.643348 20532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "abffa0d26b284d269ad4d60b9241b6f5"
format_stamp: "Formatted at 2026-06-20 14:49:49 on dist-test-slave-65tz"
server_key: "692e88d91a65fa602f3d4eeea80a088d"
server_key_iv: "0fd6cf72645865ce911832a14c8028a1"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:49.650208 20532 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 14:49:49.675364 20548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:49.677117 20532 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:49:49.677253 20532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "abffa0d26b284d269ad4d60b9241b6f5"
format_stamp: "Formatted at 2026-06-20 14:49:49 on dist-test-slave-65tz"
server_key: "692e88d91a65fa602f3d4eeea80a088d"
server_key_iv: "0fd6cf72645865ce911832a14c8028a1"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:49.677389 20532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:49.697592 20532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:49.702353 20532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:49.702615 20532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:49.703358 20532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:49:49.704479 20532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:49:49.704597 20532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:49.704708 20532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:49:49.704794 20532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:49.719185 20532 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:34763
I20260620 14:49:49.720623 20532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:49:49.723284 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 20532
I20260620 14:49:49.723455 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:49:49.723912 9079 external_mini_cluster.cc:1428] Setting key 4304a2f3304fd04a051764c4822022a7
I20260620 14:49:49.722596 20661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:34763 every 8 connection(s)
Jun 20 14:49:49 dist-test-slave-65tz krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966989, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.195@KRBTEST.COM; defaulting to no policy
I20260620 14:49:49.750770 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:49.726498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:55545 (local address 127.8.221.254:42163)
0620 14:49:49.726614 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:49.726619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:49.730335 (+ 3716us) server_negotiation.cc:407] Connection header received
0620 14:49:49.731625 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:49.731629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:49.731680 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:49.731731 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:49.733422 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:49.734293 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:49.736981 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:49.737335 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:49.737407 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:49.746484 (+ 9077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:49.746489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:49.746492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:49.746520 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:49.748303 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:49.748957 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:49.748960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:49.748962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:49.749044 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:49.749433 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:49.749437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:49.749438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:49.749656 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:49.749721 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:49.750449 (+ 728us) server_negotiation.cc:299] Negotiation successful
0620 14:49:49.750599 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:49:49.752626 20664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:49.725384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:42163 (local address 127.8.221.194:55545)
0620 14:49:49.730175 (+ 4791us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:49.730234 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:49.731372 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:49.732165 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:49.732174 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:49.732545 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:49:49.733252 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:49.733272 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:49.734450 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:49.734453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:49.735815 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:49.735826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:49.737609 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:49.737612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:49.737835 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:49.738849 (+ 1014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:49:49.738869 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:49:49.746258 (+ 7389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:49:49.748455 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:49.748461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:49.748481 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:49.748826 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:49.749150 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:49.749153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:49.749155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:49.749328 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:49.749825 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:49:49.749832 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:49:49.750168 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 14:49:49.751243 (+ 1075us) client_negotiation.cc:240] Negotiation successful
0620 14:49:49.751589 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4546,"thread_start_us":162,"threads_started":1}
I20260620 14:49:49.754242 20662 heartbeater.cc:347] Connected to a master server at 127.8.221.254:42163
I20260620 14:49:49.754482 20662 heartbeater.cc:464] Registering TS with master...
I20260620 14:49:49.755043 20662 heartbeater.cc:511] Master 127.8.221.254:42163 requested a full tablet report, sending...
I20260620 14:49:49.756150 20305 ts_manager.cc:194] Registered new tserver with Master: abffa0d26b284d269ad4d60b9241b6f5 (127.8.221.194:34763)
I20260620 14:49:49.756862 20305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:55545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
I20260620 14:49:49.785588 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.221.212:52571/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:42163
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:46333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:49:49 dist-test-slave-65tz krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966989, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
W20260620 14:49:49.960630 20667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:49.960932 20667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:49.961019 20667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:49.965505 20667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:49.965577 20667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:49.965698 20667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:49:49.971134 20667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:46333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.221.212:52571/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:42163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:49:49.972218 20667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:49.973321 20667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:49.983191 20672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:49.983662 20673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:49.983938 20675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:49.985173 20667 server_base.cc:1034] running on GCE node
I20260620 14:49:49.986008 20667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:49.986791 20667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:49.988044 20667 hybrid_clock.cc:648] HybridClock initialized: now 1781966989988028 us; error 57 us; skew 500 ppm
Jun 20 14:49:49 dist-test-slave-65tz krb5kdc[19836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966989, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:49.991130 20667 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:49:49.992022 20667 webserver.cc:460] Webserver started at http://127.8.221.195:35471/ using document root <none> and password file <none>
I20260620 14:49:49.992658 20667 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:49.992743 20667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:49.992990 20667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:49:49 dist-test-slave-65tz krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966989, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
I20260620 14:49:50.028506 20667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "c5cca946079d4f6aa9e71b782464a855"
format_stamp: "Formatted at 2026-06-20 14:49:50 on dist-test-slave-65tz"
server_key: "970bf037500f3f5c3794eda6db44c824"
server_key_iv: "f3251e606791ed0d3654cde08f942b95"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:50.029297 20667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "c5cca946079d4f6aa9e71b782464a855"
format_stamp: "Formatted at 2026-06-20 14:49:50 on dist-test-slave-65tz"
server_key: "970bf037500f3f5c3794eda6db44c824"
server_key_iv: "f3251e606791ed0d3654cde08f942b95"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:50.041582 20667 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.007s sys 0.001s
I20260620 14:49:50.057771 20682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:50.059199 20667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:49:50.059338 20667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "c5cca946079d4f6aa9e71b782464a855"
format_stamp: "Formatted at 2026-06-20 14:49:50 on dist-test-slave-65tz"
server_key: "970bf037500f3f5c3794eda6db44c824"
server_key_iv: "f3251e606791ed0d3654cde08f942b95"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:50.059475 20667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:50.086690 20667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:50.089685 20667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:50.089915 20667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:50.090584 20667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:49:50.091459 20667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:49:50.091524 20667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:50.091637 20667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:49:50.091677 20667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:50.103760 20667 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:41305
I20260620 14:49:50.104065 20795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:41305 every 8 connection(s)
I20260620 14:49:50.105044 20667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:49:50.107977 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 20667
I20260620 14:49:50.108099 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:49:50.108426 9079 external_mini_cluster.cc:1428] Setting key bd21da1d7a2515761dbec78cf16ee20e
Jun 20 14:49:50 dist-test-slave-65tz krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966989, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:49:50.127451 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:50.109595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:59719 (local address 127.8.221.254:42163)
0620 14:49:50.109774 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:50.109779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:50.111121 (+ 1342us) server_negotiation.cc:407] Connection header received
0620 14:49:50.112493 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:50.112497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:50.112562 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:50.112629 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:50.114885 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.115752 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.117640 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.118015 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.118096 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:50.123036 (+ 4940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:50.123040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:50.123043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:50.123071 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:50.124922 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:50.125528 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:50.125532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:50.125533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:50.125622 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:50.126100 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:50.126104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:50.126105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:50.126333 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:50.126435 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:50.127146 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 14:49:50.127279 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:49:50.129391 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:50.108265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:42163 (local address 127.8.221.195:59719)
0620 14:49:50.110904 (+ 2639us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:50.110973 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:50.112196 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:50.113169 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:50.113180 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:50.113630 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:49:50.114664 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:50.114692 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.115979 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.115984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:50.117478 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:50.117494 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.118324 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.118327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:50.118535 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:50.119384 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:49:50.119407 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:49:50.122837 (+ 3430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:49:50.125053 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:50.125059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:50.125077 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:50.125410 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:50.125737 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:50.125739 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:50.125742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:50.125983 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:50.126509 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:49:50.126516 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:49:50.126856 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 14:49:50.127942 (+ 1086us) client_negotiation.cc:240] Negotiation successful
0620 14:49:50.128288 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2353,"thread_start_us":189,"threads_started":1}
I20260620 14:49:50.130872 20796 heartbeater.cc:347] Connected to a master server at 127.8.221.254:42163
I20260620 14:49:50.131273 20796 heartbeater.cc:464] Registering TS with master...
I20260620 14:49:50.132028 20796 heartbeater.cc:511] Master 127.8.221.254:42163 requested a full tablet report, sending...
I20260620 14:49:50.133507 20305 ts_manager.cc:194] Registered new tserver with Master: c5cca946079d4f6aa9e71b782464a855 (127.8.221.195:41305)
I20260620 14:49:50.134363 20305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:59719
I20260620 14:49:50.134593 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:49:50.146389 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:50.136924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.8.221.254:42163)
0620 14:49:50.137056 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:50.137061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:50.137324 (+ 263us) server_negotiation.cc:407] Connection header received
0620 14:49:50.137447 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:50.137453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:50.137528 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:50.137639 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:50.138707 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.139498 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.140596 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.140945 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.141075 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:50.143245 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:50.143250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:50.143252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:50.143292 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:50.144736 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:50.145156 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:50.145160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:50.145162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:50.145238 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:50.145500 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:50.145505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:50.145506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:50.145696 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:50.145862 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:50.146093 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 14:49:50.146218 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:49:50.149542 20305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:49:50.152988 20305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Jun 20 14:49:50 dist-test-slave-65tz krb5kdc[19836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966989, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
I20260620 14:49:50.188956 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:50.163456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:34763 (local address 127.0.0.1:58624)
0620 14:49:50.165468 (+ 2012us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:50.165519 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:50.165649 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:50.168414 (+ 2765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:50.168421 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:50.168449 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:50.168640 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:50.168647 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.178797 (+ 10150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.178801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:50.181490 (+ 2689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:50.181504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.188147 (+ 6643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.188153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:50.188377 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:50.188441 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 14:49:50.188562 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 14:49:50.188763 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1813,"thread_start_us":117,"threads_started":1}
I20260620 14:49:50.190768 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:50.163592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58624 (local address 127.8.221.194:34763)
0620 14:49:50.167805 (+ 4213us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:50.167818 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:50.167847 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:49:50.167942 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:50.167949 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:50.168203 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:50.168807 (+ 604us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:50.169738 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.177863 (+ 8125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.181716 (+ 3853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.184911 (+ 3195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.189779 (+ 4868us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:50.190031 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:50.190113 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:49:50.190276 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4075,"thread_start_us":140,"threads_started":1}
I20260620 14:49:50.194468 20595 tablet_service.cc:1467] Processing CreateTablet for tablet 4de204c4aa8b427d9f5ed926e4f6a414 (DEFAULT_TABLE table=test-table [id=88509db92552412b83614ca7c6962825]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:49:50.195829 20595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4de204c4aa8b427d9f5ed926e4f6a414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:49:50.204370 20815 tablet_bootstrap.cc:492] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5: Bootstrap starting.
I20260620 14:49:50.207294 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:50.164809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:43531 (local address 127.0.0.1:54140)
0620 14:49:50.178440 (+ 13631us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:50.178453 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:50.178665 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:50.182870 (+ 4205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:50.182873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:50.182891 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:50.183021 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:50.183027 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.190490 (+ 7463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.190494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:50.203627 (+ 13133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:50.203646 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.206687 (+ 3041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.206704 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:50.206864 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:50.206908 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 14:49:50.207005 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 14:49:50.207130 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13581,"thread_start_us":96,"threads_started":1}
I20260620 14:49:50.207894 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:50.165236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54140 (local address 127.8.221.193:43531)
0620 14:49:50.166099 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:50.166106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:50.178573 (+ 12467us) server_negotiation.cc:407] Connection header received
0620 14:49:50.182137 (+ 3564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:50.182146 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:50.182478 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:50.182655 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:50.185089 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.190322 (+ 5233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.204564 (+ 14242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.205782 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.205926 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:50.206124 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:50.207546 (+ 1422us) server_negotiation.cc:299] Negotiation successful
0620 14:49:50.207727 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":124,"threads_started":1}
I20260620 14:49:50.208695 20815 tablet_bootstrap.cc:654] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5: Neither blocks nor log segments found. Creating new log.
I20260620 14:49:50.209679 20815 log.cc:824] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:50.211274 20460 tablet_service.cc:1467] Processing CreateTablet for tablet 4de204c4aa8b427d9f5ed926e4f6a414 (DEFAULT_TABLE table=test-table [id=88509db92552412b83614ca7c6962825]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:49:50.211928 20815 tablet_bootstrap.cc:492] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5: No bootstrap required, opened a new log
I20260620 14:49:50.212118 20815 ts_tablet_manager.cc:1397] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 14:49:50.212397 20460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4de204c4aa8b427d9f5ed926e4f6a414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:49:50.216282 20815 raft_consensus.cc:348] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abffa0d26b284d269ad4d60b9241b6f5" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34763 } } peers { permanent_uuid: "c5cca946079d4f6aa9e71b782464a855" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 41305 } } peers { permanent_uuid: "08e72c0b004c4f488216fd594e434ebc" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43531 } }
I20260620 14:49:50.216547 20815 raft_consensus.cc:374] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:49:50.216598 20815 raft_consensus.cc:729] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abffa0d26b284d269ad4d60b9241b6f5, State: Initialized, Role: FOLLOWER
I20260620 14:49:50.217103 20815 consensus_queue.cc:260] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abffa0d26b284d269ad4d60b9241b6f5" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34763 } } peers { permanent_uuid: "c5cca946079d4f6aa9e71b782464a855" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 41305 } } peers { permanent_uuid: "08e72c0b004c4f488216fd594e434ebc" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43531 } }
I20260620 14:49:50.218416 20815 ts_tablet_manager.cc:1428] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:49:50.218837 20662 heartbeater.cc:503] Master 127.8.221.254:42163 was elected leader, sending a full tablet report...
I20260620 14:49:50.221460 20819 tablet_bootstrap.cc:492] T 4de204c4aa8b427d9f5ed926e4f6a414 P 08e72c0b004c4f488216fd594e434ebc: Bootstrap starting.
W20260620 14:49:50.224215 20663 tablet.cc:2369] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:49:50.225993 20819 tablet_bootstrap.cc:654] T 4de204c4aa8b427d9f5ed926e4f6a414 P 08e72c0b004c4f488216fd594e434ebc: Neither blocks nor log segments found. Creating new log.
I20260620 14:49:50.226080 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:50.164421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.8.221.195:41305)
0620 14:49:50.165174 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:50.165189 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:50.178300 (+ 13111us) server_negotiation.cc:407] Connection header received
0620 14:49:50.183227 (+ 4927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:50.183234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:50.183426 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:50.183560 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:50.188094 (+ 4534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.189281 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.222409 (+ 33128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.223515 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.223625 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:50.223806 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:50.225721 (+ 1915us) server_negotiation.cc:299] Negotiation successful
0620 14:49:50.225886 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":161,"threads_started":1}
I20260620 14:49:50.226945 20819 log.cc:824] T 4de204c4aa8b427d9f5ed926e4f6a414 P 08e72c0b004c4f488216fd594e434ebc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:50.227031 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:50.163530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:41305 (local address 127.0.0.1:44982)
0620 14:49:50.178166 (+ 14636us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:50.178182 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:50.181890 (+ 3708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:50.183758 (+ 1868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:50.183761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:50.183779 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:50.183878 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:50.183884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.191075 (+ 7191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.191079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:50.206337 (+ 15258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:50.206350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.225417 (+ 19067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.225421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:50.225545 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:50.225572 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:49:50.226594 (+ 1022us) client_negotiation.cc:240] Negotiation successful
0620 14:49:50.226859 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14540,"thread_start_us":133,"threads_started":1}
I20260620 14:49:50.230898 20724 tablet_service.cc:1467] Processing CreateTablet for tablet 4de204c4aa8b427d9f5ed926e4f6a414 (DEFAULT_TABLE table=test-table [id=88509db92552412b83614ca7c6962825]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:49:50.232283 20724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4de204c4aa8b427d9f5ed926e4f6a414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:49:50.241055 20819 tablet_bootstrap.cc:492] T 4de204c4aa8b427d9f5ed926e4f6a414 P 08e72c0b004c4f488216fd594e434ebc: No bootstrap required, opened a new log
I20260620 14:49:50.241322 20819 ts_tablet_manager.cc:1397] T 4de204c4aa8b427d9f5ed926e4f6a414 P 08e72c0b004c4f488216fd594e434ebc: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20260620 14:49:50.244038 20821 tablet_bootstrap.cc:492] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855: Bootstrap starting.
I20260620 14:49:50.247597 20821 tablet_bootstrap.cc:654] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855: Neither blocks nor log segments found. Creating new log.
I20260620 14:49:50.248668 20821 log.cc:824] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:50.248800 20819 raft_consensus.cc:348] T 4de204c4aa8b427d9f5ed926e4f6a414 P 08e72c0b004c4f488216fd594e434ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abffa0d26b284d269ad4d60b9241b6f5" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34763 } } peers { permanent_uuid: "c5cca946079d4f6aa9e71b782464a855" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 41305 } } peers { permanent_uuid: "08e72c0b004c4f488216fd594e434ebc" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43531 } }
I20260620 14:49:50.249717 20819 raft_consensus.cc:374] T 4de204c4aa8b427d9f5ed926e4f6a414 P 08e72c0b004c4f488216fd594e434ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:49:50.249881 20819 raft_consensus.cc:729] T 4de204c4aa8b427d9f5ed926e4f6a414 P 08e72c0b004c4f488216fd594e434ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08e72c0b004c4f488216fd594e434ebc, State: Initialized, Role: FOLLOWER
I20260620 14:49:50.250597 20819 consensus_queue.cc:260] T 4de204c4aa8b427d9f5ed926e4f6a414 P 08e72c0b004c4f488216fd594e434ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abffa0d26b284d269ad4d60b9241b6f5" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34763 } } peers { permanent_uuid: "c5cca946079d4f6aa9e71b782464a855" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 41305 } } peers { permanent_uuid: "08e72c0b004c4f488216fd594e434ebc" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43531 } }
I20260620 14:49:50.251169 20821 tablet_bootstrap.cc:492] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855: No bootstrap required, opened a new log
I20260620 14:49:50.251395 20821 ts_tablet_manager.cc:1397] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 14:49:50.251965 20819 ts_tablet_manager.cc:1428] T 4de204c4aa8b427d9f5ed926e4f6a414 P 08e72c0b004c4f488216fd594e434ebc: Time spent starting tablet: real 0.011s user 0.003s sys 0.004s
I20260620 14:49:50.252378 20526 heartbeater.cc:503] Master 127.8.221.254:42163 was elected leader, sending a full tablet report...
I20260620 14:49:50.257138 20821 raft_consensus.cc:348] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abffa0d26b284d269ad4d60b9241b6f5" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34763 } } peers { permanent_uuid: "c5cca946079d4f6aa9e71b782464a855" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 41305 } } peers { permanent_uuid: "08e72c0b004c4f488216fd594e434ebc" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43531 } }
I20260620 14:49:50.258211 20821 raft_consensus.cc:374] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:49:50.258354 20821 raft_consensus.cc:729] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cca946079d4f6aa9e71b782464a855, State: Initialized, Role: FOLLOWER
I20260620 14:49:50.259945 20821 consensus_queue.cc:260] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abffa0d26b284d269ad4d60b9241b6f5" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34763 } } peers { permanent_uuid: "c5cca946079d4f6aa9e71b782464a855" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 41305 } } peers { permanent_uuid: "08e72c0b004c4f488216fd594e434ebc" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43531 } }
I20260620 14:49:50.261222 20821 ts_tablet_manager.cc:1428] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260620 14:49:50.261673 20796 heartbeater.cc:503] Master 127.8.221.254:42163 was elected leader, sending a full tablet report...
W20260620 14:49:50.329973 20527 tablet.cc:2369] T 4de204c4aa8b427d9f5ed926e4f6a414 P 08e72c0b004c4f488216fd594e434ebc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:49:50.364645 20797 tablet.cc:2369] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:49:50.487001 20817 raft_consensus.cc:482] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:49:50.487336 20817 raft_consensus.cc:504] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abffa0d26b284d269ad4d60b9241b6f5" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34763 } } peers { permanent_uuid: "c5cca946079d4f6aa9e71b782464a855" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 41305 } } peers { permanent_uuid: "08e72c0b004c4f488216fd594e434ebc" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43531 } }
I20260620 14:49:50.488970 20817 leader_election.cc:288] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5cca946079d4f6aa9e71b782464a855 (127.8.221.195:41305), 08e72c0b004c4f488216fd594e434ebc (127.8.221.193:43531)
I20260620 14:49:50.495826 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:50.489441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:50309 (local address 127.8.221.195:41305)
0620 14:49:50.489633 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:50.489642 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:50.489968 (+ 326us) server_negotiation.cc:407] Connection header received
0620 14:49:50.490249 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:50.490253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:50.490340 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:50.490423 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:50.490952 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.491883 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.494167 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.494870 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.494983 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:50.495050 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:50.495564 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 14:49:50.495654 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 14:49:50.495826 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:50.490038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:46189 (local address 127.8.221.193:43531)
0620 14:49:50.490239 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:50.490245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:50.490608 (+ 363us) server_negotiation.cc:407] Connection header received
0620 14:49:50.490884 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:50.490887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:50.490953 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:50.491021 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:50.491446 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.492224 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.494162 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.494870 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.494994 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:50.495050 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:50.495564 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 14:49:50.495654 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:49:50.496383 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:50.489751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:43531 (local address 127.8.221.194:46189)
0620 14:49:50.490513 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:50.490528 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:50.490736 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:50.491168 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:50.491172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:50.491192 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:50.491310 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:50.491316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.492359 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.492362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:50.494001 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:50.494011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.495182 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.495185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:50.495332 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:50.495426 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 14:49:50.496076 (+ 650us) client_negotiation.cc:240] Negotiation successful
0620 14:49:50.496196 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"mutex_wait_us":172,"spinlock_wait_cycles":54528,"thread_start_us":453,"threads_started":1}
I20260620 14:49:50.496678 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:50.489281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:41305 (local address 127.8.221.194:50309)
0620 14:49:50.489855 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:50.489880 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:50.490079 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:50.490636 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:50.490640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:50.490667 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:50.490816 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:50.490828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.492337 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.492341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:50.493999 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:50.494011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.495167 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.495170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:50.495317 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:50.495422 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 14:49:50.496407 (+ 985us) client_negotiation.cc:240] Negotiation successful
0620 14:49:50.496525 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":173,"thread_start_us":113,"threads_started":1}
I20260620 14:49:50.497395 20742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de204c4aa8b427d9f5ed926e4f6a414" candidate_uuid: "abffa0d26b284d269ad4d60b9241b6f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cca946079d4f6aa9e71b782464a855" is_pre_election: true
I20260620 14:49:50.497371 20461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de204c4aa8b427d9f5ed926e4f6a414" candidate_uuid: "abffa0d26b284d269ad4d60b9241b6f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08e72c0b004c4f488216fd594e434ebc" is_pre_election: true
I20260620 14:49:50.497835 20461 raft_consensus.cc:2454] T 4de204c4aa8b427d9f5ed926e4f6a414 P 08e72c0b004c4f488216fd594e434ebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abffa0d26b284d269ad4d60b9241b6f5 in term 0.
I20260620 14:49:50.497857 20742 raft_consensus.cc:2454] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abffa0d26b284d269ad4d60b9241b6f5 in term 0.
I20260620 14:49:50.498481 20552 leader_election.cc:302] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08e72c0b004c4f488216fd594e434ebc, abffa0d26b284d269ad4d60b9241b6f5; no voters:
I20260620 14:49:50.498817 20817 raft_consensus.cc:2781] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:49:50.498914 20817 raft_consensus.cc:482] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:49:50.498973 20817 raft_consensus.cc:3037] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:49:50.500259 20817 raft_consensus.cc:504] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abffa0d26b284d269ad4d60b9241b6f5" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34763 } } peers { permanent_uuid: "c5cca946079d4f6aa9e71b782464a855" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 41305 } } peers { permanent_uuid: "08e72c0b004c4f488216fd594e434ebc" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43531 } }
I20260620 14:49:50.500660 20817 leader_election.cc:288] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [CANDIDATE]: Term 1 election: Requested vote from peers c5cca946079d4f6aa9e71b782464a855 (127.8.221.195:41305), 08e72c0b004c4f488216fd594e434ebc (127.8.221.193:43531)
I20260620 14:49:50.501123 20461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de204c4aa8b427d9f5ed926e4f6a414" candidate_uuid: "abffa0d26b284d269ad4d60b9241b6f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08e72c0b004c4f488216fd594e434ebc"
I20260620 14:49:50.501271 20742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de204c4aa8b427d9f5ed926e4f6a414" candidate_uuid: "abffa0d26b284d269ad4d60b9241b6f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5cca946079d4f6aa9e71b782464a855"
I20260620 14:49:50.501286 20461 raft_consensus.cc:3037] T 4de204c4aa8b427d9f5ed926e4f6a414 P 08e72c0b004c4f488216fd594e434ebc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:49:50.501410 20742 raft_consensus.cc:3037] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:49:50.502792 20461 raft_consensus.cc:2454] T 4de204c4aa8b427d9f5ed926e4f6a414 P 08e72c0b004c4f488216fd594e434ebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abffa0d26b284d269ad4d60b9241b6f5 in term 1.
I20260620 14:49:50.502856 20742 raft_consensus.cc:2454] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abffa0d26b284d269ad4d60b9241b6f5 in term 1.
I20260620 14:49:50.503271 20552 leader_election.cc:302] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08e72c0b004c4f488216fd594e434ebc, abffa0d26b284d269ad4d60b9241b6f5; no voters:
I20260620 14:49:50.503605 20817 raft_consensus.cc:2781] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:49:50.503890 20817 raft_consensus.cc:686] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [term 1 LEADER]: Becoming Leader. State: Replica: abffa0d26b284d269ad4d60b9241b6f5, State: Running, Role: LEADER
I20260620 14:49:50.504297 20817 consensus_queue.cc:237] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abffa0d26b284d269ad4d60b9241b6f5" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34763 } } peers { permanent_uuid: "c5cca946079d4f6aa9e71b782464a855" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 41305 } } peers { permanent_uuid: "08e72c0b004c4f488216fd594e434ebc" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43531 } }
I20260620 14:49:50.508586 20305 catalog_manager.cc:5495] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to abffa0d26b284d269ad4d60b9241b6f5 (127.8.221.194). New cstate: current_term: 1 leader_uuid: "abffa0d26b284d269ad4d60b9241b6f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abffa0d26b284d269ad4d60b9241b6f5" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5cca946079d4f6aa9e71b782464a855" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 41305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08e72c0b004c4f488216fd594e434ebc" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43531 } health_report { overall_health: UNKNOWN } } }
I20260620 14:49:50.539896 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:50.534209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58640 (local address 127.8.221.194:34763)
0620 14:49:50.534388 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:50.534395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:50.534418 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:49:50.534534 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:50.534538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:50.534632 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:50.534768 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:49:50.535168 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.536067 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.537347 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:50.537701 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:50.537842 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:50.538373 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:49:50.539098 (+ 725us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:49:50.539242 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:50.539626 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 14:49:50.539704 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:49:50.548591 20461 raft_consensus.cc:1261] T 4de204c4aa8b427d9f5ed926e4f6a414 P 08e72c0b004c4f488216fd594e434ebc [term 1 FOLLOWER]: Refusing update from remote peer abffa0d26b284d269ad4d60b9241b6f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:49:50.549019 20742 raft_consensus.cc:1261] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855 [term 1 FOLLOWER]: Refusing update from remote peer abffa0d26b284d269ad4d60b9241b6f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:49:50.549876 20817 consensus_queue.cc:1035] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08e72c0b004c4f488216fd594e434ebc" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:49:50.550153 20834 consensus_queue.cc:1035] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5cca946079d4f6aa9e71b782464a855" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:49:50.566629 20839 mvcc.cc:205] Tried to move back new op lower bound from 7298936793277071360 to 7298936793108557824. Current Snapshot: MvccSnapshot[applied={T|T < 7298936793277071360}]
I20260620 14:49:50.570749 20838 mvcc.cc:205] Tried to move back new op lower bound from 7298936793277071360 to 7298936793108557824. Current Snapshot: MvccSnapshot[applied={T|T < 7298936793277071360}]
I20260620 14:49:50.573345 20309 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:49:50.573545 20309 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:49:50.576702 20309 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4889d5ee0a0846e7a5ea3fa237390d5c: Sending DeleteTablet for 3 replicas of tablet 4de204c4aa8b427d9f5ed926e4f6a414
I20260620 14:49:50.577482 20595 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de204c4aa8b427d9f5ed926e4f6a414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:49:50 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:58624
I20260620 14:49:50.577682 20724 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de204c4aa8b427d9f5ed926e4f6a414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:49:50 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:44982
I20260620 14:49:50.578671 20460 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de204c4aa8b427d9f5ed926e4f6a414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:49:50 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:54140
I20260620 14:49:50.579245 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 20393
I20260620 14:49:50.590018 20847 tablet_replica.cc:330] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855: stopping tablet replica
I20260620 14:49:50.590515 20847 raft_consensus.cc:2229] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:49:50.590999 20847 raft_consensus.cc:2258] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:49:50.592563 20847 ts_tablet_manager.cc:1905] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:49:50.594388 20846 tablet_replica.cc:330] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5: stopping tablet replica
I20260620 14:49:50.594962 20846 raft_consensus.cc:2229] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:49:50.595602 20847 ts_tablet_manager.cc:1918] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:49:50.595697 20846 raft_consensus.cc:2258] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:49:50.600430 20846 ts_tablet_manager.cc:1905] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:49:50.602324 20847 log.cc:1196] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/4de204c4aa8b427d9f5ed926e4f6a414
I20260620 14:49:50.602823 20847 ts_tablet_manager.cc:1939] T 4de204c4aa8b427d9f5ed926e4f6a414 P c5cca946079d4f6aa9e71b782464a855: Deleting consensus metadata
I20260620 14:49:50.604034 20846 ts_tablet_manager.cc:1918] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:49:50.604456 20299 catalog_manager.cc:4841] TS c5cca946079d4f6aa9e71b782464a855 (127.8.221.195:41305): tablet 4de204c4aa8b427d9f5ed926e4f6a414 (table test-table [id=88509db92552412b83614ca7c6962825]) successfully deleted
W20260620 14:49:50.605125 20299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.221.193:43531 (error 108)
I20260620 14:49:50.606091 20846 log.cc:1196] T 4de204c4aa8b427d9f5ed926e4f6a414 P abffa0d26b284d269ad4d60b9241b6f5: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/4de204c4aa8b427d9f5ed926e4f6a414
I20260620 14:49:50.606400 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 20532
I20260620 14:49:50.608287 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:50.607821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:43531 (local address 127.0.0.1:54154)
0620 14:49:50.607997 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:50.608156 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.193:43531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 14:49:50.608522 20299 catalog_manager.cc:4584] TS 08e72c0b004c4f488216fd594e434ebc (127.8.221.193:43531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4de204c4aa8b427d9f5ed926e4f6a414: Network error: Client connection negotiation failed: client connection to 127.8.221.193:43531: connect: Connection refused (error 111)
I20260620 14:49:50.616550 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:50.616265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:34763 (local address 127.0.0.1:58642)
0620 14:49:50.616381 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:50.616441 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.194:34763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 14:49:50.616688 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 20667
I20260620 14:49:50.623287 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 20280
I20260620 14:49:50.632961 9079 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:49:50.971076 9079 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:49:50.971215 9079 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:49:51.315716 9079 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:49:51.315 UTC [19866] LOG: received smart shutdown request
2026-06-20 14:49:51.319 UTC [19866] LOG: background worker "logical replication launcher" (PID 19875) exited with exit code 1
2026-06-20 14:49:51.319 UTC [19870] LOG: shutting down
2026-06-20 14:49:51.487 UTC [19866] LOG: database system is shut down
2026-06-20T14:49:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50224 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:49:51 dist-test-slave-65tz krb5kdc[20858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:49:51 dist-test-slave-65tz krb5kdc[20858](info): set up 2 sockets
Jun 20 14:49:51 dist-test-slave-65tz krb5kdc[20858](info): commencing operation
krb5kdc: starting...
W20260620 14:49:52.670326 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:49:52 dist-test-slave-65tz krb5kdc[20858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:49:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:49:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.221.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.221.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.221.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/rangeradmin_127.8.221.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.221.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/rangeradmin_127.8.221.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.221.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.221.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.221.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/rangerlookup_127.8.221.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.221.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/rangerlookup_127.8.221.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.221.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.221.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.221.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/HTTP_127.8.221.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.221.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/HTTP_127.8.221.212@KRBTEST.COM.keytab.
I20260620 14:49:52.864449 9079 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQhiojW/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/postgres -l logfile start
2026-06-20 14:49:55.398 UTC [20888] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:49:55.398 UTC [20888] LOG: listening on IPv4 address "127.8.221.212", port 33803
2026-06-20 14:49:55.402 UTC [20888] LOG: listening on Unix socket "/tmp/.s.PGSQL.33803"
2026-06-20 14:49:55.440 UTC [20891] LOG: database system was shut down at 2026-06-20 14:49:53 UTC
2026-06-20 14:49:55.445 UTC [20888] LOG: database system is ready to accept connections
I20260620 14:49:56.406083 9079 mini_postgres.cc:96] Postgres bound to 33803
2026-06-20 14:49:56.413 UTC [20899] FATAL: database "slave" does not exist
127.8.221.212:33803 - accepting connections
I20260620 14:49:56.413879 9079 mini_ranger.cc:161] Starting Ranger...
I20260620 14:49:56.430519 9079 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:49:56.925469 9079 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:49:56.925576 9079 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ranger-admin
2026-06-20 14:49:57,223 [I] DB FLAVOR :POSTGRES
2026-06-20 14:49:57,224 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:49:57,224 [I] Checking connection..
2026-06-20 14:49:57,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:49:57,498 [I] Checking connection passed.
2026-06-20 14:49:57,498 [I] --------- Verifying version history table ---------
2026-06-20 14:49:57,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:49:57,782 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:49:57,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:49:58,068 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:49:58,068 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:49:58,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:49:58.343 UTC [20963] WARNING: there is no transaction in progress
2026-06-20 14:49:58,351 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:49:58,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:49:58,661 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:49:58,661 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:49:58,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:49:58,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:49:59,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-65tz.c.gcp-upstream.internal','N') ;"
2026-06-20 14:49:59,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:49:59,703 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:49:59,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:49:59,966 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:49:59,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:50:00,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:50:00,517 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:50:00,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:50:01.556 UTC [21071] WARNING: there is no transaction in progress
2026-06-20 14:50:01.573 UTC [21071] WARNING: there is no transaction in progress
2026-06-20 14:50:01.592 UTC [21071] WARNING: there is no transaction in progress
2026-06-20 14:50:01.609 UTC [21071] WARNING: there is no transaction in progress
2026-06-20 14:50:01.630 UTC [21071] WARNING: there is no transaction in progress
2026-06-20 14:50:01.653 UTC [21071] WARNING: there is no transaction in progress
2026-06-20 14:50:01.715 UTC [21071] WARNING: there is no transaction in progress
2026-06-20 14:50:01.723 UTC [21071] WARNING: there is no transaction in progress
2026-06-20 14:50:01.739 UTC [21071] WARNING: there is no transaction in progress
2026-06-20 14:50:01.750 UTC [21071] WARNING: there is no transaction in progress
2026-06-20 14:50:01.762 UTC [21071] WARNING: there is no transaction in progress
2026-06-20 14:50:01.778 UTC [21071] WARNING: there is no transaction in progress
2026-06-20 14:50:01.789 UTC [21071] WARNING: there is no transaction in progress
2026-06-20 14:50:01.796 UTC [21071] WARNING: there is no transaction in progress
2026-06-20 14:50:01.803 UTC [21071] WARNING: there is no transaction in progress
2026-06-20 14:50:01.810 UTC [21071] WARNING: there is no transaction in progress
2026-06-20 14:50:02,542 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:50:02,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:50:02,814 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:50:02,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:50:03,115 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:50:03,116 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:50:03,415 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:50:03,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:50:03,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:50:04,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-65tz.c.gcp-upstream.internal';"
2026-06-20 14:50:04,286 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:50:04,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:50:04,540 [I] DB_PATCHES have already been applied
I20260620 14:50:04.545781 9079 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskQhiojW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:50:04.545946 9079 mini_ranger.cc:193] Using host: 127.8.221.212
I20260620 14:50:04.550196 9079 mini_ranger.cc:227] Ranger admin URL: http://127.8.221.212:39761
Jun 20, 2026 2:50:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:50:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:50:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:50:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:50:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:50:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:50:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:50:05.762709 9079 mini_ranger.cc:160] Time spent starting Ranger: real 9.349s user 0.001s sys 0.011s
Jun 20, 2026 2:50:27 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:50:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 2:50:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:50:27 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:50:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:50:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:50:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:50:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:50:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:50:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:50:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:50:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:50:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:50:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:50:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:50:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:50:27 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:50:28 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:50:29.210574 9079 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.221.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.221.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.221.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/rangerkms_127.8.221.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.221.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/rangerkms_127.8.221.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.221.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.221.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.221.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/HTTP_127.8.221.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.221.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/HTTP_127.8.221.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:50:29 dist-test-slave-65tz krb5kdc[20858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967029, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:50:29.320518 9079 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:50:29.332150 9079 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:50:30.252672 9079 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:50:30.252779 9079 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ranger-kms
I20260620 14:50:30.252801 9079 mini_ranger_kms.cc:227] Using postgres at 127.8.221.212:33803
2026-06-20 14:50:30,423 [I] DB FLAVOR :POSTGRES
2026-06-20 14:50:30,424 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:50:30,424 [I] Checking connection
2026-06-20 14:50:30,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:50:30,709 [I] connection success
2026-06-20 14:50:30,709 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:50:30,709 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:50:30,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:50:30,992 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:50:30,993 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:50:30,993 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:50:30,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.221.212:33803/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:50:31,304 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:50:31.856978 9079 mini_ranger_kms.cc:310] Created kms service
I20260620 14:50:32.079169 9079 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:50:32.128433 9079 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:50:32 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967029, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
I20260620 14:50:32.500134 9079 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:50:32.500381 9079 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQhiojW/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQhiojW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:50:32.500413 9079 mini_ranger_kms.cc:240] Using host: 127.8.221.212
I20260620 14:50:32.504015 9079 mini_ranger_kms.cc:276] Ranger KMS PID: 21252
I20260620 14:50:32.504098 9079 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.221.212:47889
Jun 20, 2026 2:50:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:50:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:50:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:50:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:50:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:50:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 14:50:34 dist-test-slave-65tz krb5kdc[20858](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967034, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.221.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:34.926954 9079 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.606s user 0.011s sys 0.144s
I20260620 14:50:34.927105 9079 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:50:34.927176 9079 mini_ranger_kms.cc:401] 127.8.221.212:47889/kms/v1/keys
Jun 20 14:50:35 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967034, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.221.212@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
Jun 20 14:50:35 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967034, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.221.212@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
Jun 20 14:50:37 dist-test-slave-65tz krb5kdc[20858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:50:38.002674 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:42215
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:42215
--ranger_config_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.221.212:47889/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:38.147264 21302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:38.147629 21302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:38.147675 21302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:38.152009 21302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:38.152091 21302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:38.152112 21302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:38.152128 21302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:38.152144 21302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:38.158305 21302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.221.212:47889/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:42215
--ranger_config_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:42215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:38.160060 21302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:38.161392 21302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:50:38.171960 21302 server_base.cc:1034] running on GCE node
W20260620 14:50:38.172013 21310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:38.172003 21308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:38.172230 21307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:38.173035 21302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:38.174324 21302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:38.175532 21302 hybrid_clock.cc:648] HybridClock initialized: now 1781967038175448 us; error 112 us; skew 500 ppm
Jun 20 14:50:38 dist-test-slave-65tz krb5kdc[20858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967038, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:38.181496 21302 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:50:38.182502 21302 webserver.cc:460] Webserver started at http://127.8.221.254:34807/ using document root <none> and password file <none>
I20260620 14:50:38.183120 21302 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:38.183203 21302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:38.183426 21302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:50:38 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967038, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
I20260620 14:50:38.241077 21302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc"
format_stamp: "Formatted at 2026-06-20 14:50:38 on dist-test-slave-65tz"
server_key: "b8db2f5fdff3ace8cfd422c3ee6d054f"
server_key_iv: "a30268be6ad094af3fa44c4db0de9e90"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:38.241882 21302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc"
format_stamp: "Formatted at 2026-06-20 14:50:38 on dist-test-slave-65tz"
server_key: "b8db2f5fdff3ace8cfd422c3ee6d054f"
server_key_iv: "a30268be6ad094af3fa44c4db0de9e90"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:38.247036 21302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260620 14:50:38.265125 21317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:38.267138 21302 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 14:50:38.267352 21302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc"
format_stamp: "Formatted at 2026-06-20 14:50:38 on dist-test-slave-65tz"
server_key: "b8db2f5fdff3ace8cfd422c3ee6d054f"
server_key_iv: "a30268be6ad094af3fa44c4db0de9e90"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:38.267517 21302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:38.282958 21302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:38.286469 21302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:38.286984 21302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:38.296454 21302 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:42215
I20260620 14:50:38.297022 21369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:42215 every 8 connection(s)
I20260620 14:50:38.298079 21302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:38.302014 21370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:38.302685 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 21302
I20260620 14:50:38.302865 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:50:38.303241 9079 external_mini_cluster.cc:1428] Setting key 92f10575f5d986c2e5fe08e9c4472f65
I20260620 14:50:38.309438 21370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc: Bootstrap starting.
I20260620 14:50:38.312180 21370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:38.312927 21370 log.cc:824] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:50:38 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:50:38.314944 21370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc: No bootstrap required, opened a new log
I20260620 14:50:38.318272 21370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 42215 } }
I20260620 14:50:38.318600 21370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:38.318682 21370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b5c7d3d9ba24fbf960b9fa633f8f5bc, State: Initialized, Role: FOLLOWER
I20260620 14:50:38.319146 21370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 42215 } }
I20260620 14:50:38.319293 21370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:38.319377 21370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:38.319499 21370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:38.320417 21370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 42215 } }
I20260620 14:50:38.320776 21370 leader_election.cc:302] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b5c7d3d9ba24fbf960b9fa633f8f5bc; no voters:
I20260620 14:50:38.321133 21370 leader_election.cc:288] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:38.321198 21375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:38.321406 21375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [term 1 LEADER]: Becoming Leader. State: Replica: 1b5c7d3d9ba24fbf960b9fa633f8f5bc, State: Running, Role: LEADER
I20260620 14:50:38.321789 21375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 42215 } }
I20260620 14:50:38.322278 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:38.305179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33562 (local address 127.8.221.254:42215)
0620 14:50:38.305704 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:38.305724 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:38.305769 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:50:38.306716 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:38.306739 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:38.307177 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:38.307603 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:38.308587 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:38.309750 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:38.310788 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:38.311371 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:38.311572 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:38.314726 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:38.314733 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:38.314754 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:38.314789 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:38.317784 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:38.318420 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:38.318430 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:38.318442 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:38.318621 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:38.319173 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:38.319180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:38.319184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:38.319872 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:38.320144 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:38.320411 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 14:50:38.320982 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":173,"threads_started":1}
I20260620 14:50:38.322423 21370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:38.323720 21377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b5c7d3d9ba24fbf960b9fa633f8f5bc. Latest consensus state: current_term: 1 leader_uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 42215 } } }
I20260620 14:50:38.324189 21377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:38.324069 21376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 42215 } } }
I20260620 14:50:38.324533 21376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:38.324913 21370 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:50:38.325089 21388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:38.332381 21388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:38.340507 21388 catalog_manager.cc:1347] Generated new cluster ID: 55f7d27f67dd4dc6989a91152b1af781
I20260620 14:50:38.340590 21388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:38.379199 21388 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:38.380532 21388 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:38.389189 21388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc: Generated new TSK 0
I20260620 14:50:38.390091 21388 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
I20260620 14:50:39.162638 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.221.212:47889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:42215
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:39.326614 21415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:39.327142 21415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:39.327219 21415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:39.333825 21415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:39.333945 21415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:39.334044 21415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:50:39.342584 21415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.221.212:47889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:42215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:39.344149 21415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:39.345463 21415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:39.359683 21423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:39.359688 21422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:39.359892 21425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:39.361159 21415 server_base.cc:1034] running on GCE node
I20260620 14:50:39.361704 21415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:39.362687 21415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:39.363906 21415 hybrid_clock.cc:648] HybridClock initialized: now 1781967039363864 us; error 58 us; skew 500 ppm
Jun 20 14:50:39 dist-test-slave-65tz krb5kdc[20858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967039, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:39.367777 21415 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:50:39.369124 21415 webserver.cc:460] Webserver started at http://127.8.221.193:39191/ using document root <none> and password file <none>
I20260620 14:50:39.369951 21415 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:39.370059 21415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:39.370388 21415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:50:39 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967039, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
I20260620 14:50:39.394351 21415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "1d5b4aad460e40aeaee16e6d69808754"
format_stamp: "Formatted at 2026-06-20 14:50:39 on dist-test-slave-65tz"
server_key: "4fd6253e9bb1340584680ef4e047d403"
server_key_iv: "4b2c67c292c210496b11b23d089fe27b"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:39.395282 21415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "1d5b4aad460e40aeaee16e6d69808754"
format_stamp: "Formatted at 2026-06-20 14:50:39 on dist-test-slave-65tz"
server_key: "4fd6253e9bb1340584680ef4e047d403"
server_key_iv: "4b2c67c292c210496b11b23d089fe27b"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:39.402828 21415 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 14:50:39.431249 21433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:39.433854 21415 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.004s sys 0.000s
I20260620 14:50:39.434063 21415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "1d5b4aad460e40aeaee16e6d69808754"
format_stamp: "Formatted at 2026-06-20 14:50:39 on dist-test-slave-65tz"
server_key: "4fd6253e9bb1340584680ef4e047d403"
server_key_iv: "4b2c67c292c210496b11b23d089fe27b"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:39.434259 21415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:39.462306 21415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:39.466741 21415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:39.467064 21415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:39.467944 21415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:39.469184 21415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:39.469290 21415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:39.469431 21415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:39.469520 21415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:39.482335 21415 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:35573
I20260620 14:50:39.482429 21547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:35573 every 8 connection(s)
I20260620 14:50:39.483867 21415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:50:39.486330 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 21415
I20260620 14:50:39.486452 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:50:39.486799 9079 external_mini_cluster.cc:1428] Setting key 65fc0f14b19b1e2fae4224deca6dfe29
Jun 20 14:50:39 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967039, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.194@KRBTEST.COM" created.
I20260620 14:50:39.506458 21551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:39.488448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:41003 (local address 127.8.221.254:42215)
0620 14:50:39.488774 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:39.488779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:39.490753 (+ 1974us) server_negotiation.cc:407] Connection header received
0620 14:50:39.492172 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:39.492177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:39.492257 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:39.492379 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:39.494332 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:39.495237 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:39.497062 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:39.497422 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:39.497508 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:39.501137 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:39.501141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:39.501144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:39.501173 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:39.503345 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:39.504022 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:39.504027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:39.504029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:39.504123 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:39.504561 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:39.504565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:39.504567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:39.504829 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:39.504924 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:39.506037 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0620 14:50:39.506230 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":115,"threads_started":1}
I20260620 14:50:39.509007 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:39.487064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:42215 (local address 127.8.221.193:41003)
0620 14:50:39.490543 (+ 3479us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:39.490623 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:39.491869 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:39.492794 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:39.492806 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:39.493270 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:39.494125 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:39.494150 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:39.495409 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:39.495412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:39.496915 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:39.496926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:39.497754 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:39.497759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:39.497997 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:39.498824 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:39.498848 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:39.500920 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:39.503486 (+ 2566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:39.503493 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:39.503515 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:39.503905 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:39.504242 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:39.504245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:39.504247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:39.504443 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:39.504996 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:39.505002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:39.505646 (+ 644us) client_negotiation.cc:769] Sending connection context
0620 14:50:39.507093 (+ 1447us) client_negotiation.cc:240] Negotiation successful
0620 14:50:39.507635 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2833,"thread_start_us":183,"threads_started":1}
I20260620 14:50:39.511224 21548 heartbeater.cc:347] Connected to a master server at 127.8.221.254:42215
I20260620 14:50:39.511605 21548 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:39.512358 21548 heartbeater.cc:511] Master 127.8.221.254:42215 requested a full tablet report, sending...
I20260620 14:50:39.514472 21332 ts_manager.cc:194] Registered new tserver with Master: 1d5b4aad460e40aeaee16e6d69808754 (127.8.221.193:35573)
I20260620 14:50:39.516129 21332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:41003
Jun 20 14:50:39 dist-test-slave-65tz krb5kdc[20858](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967039, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
I20260620 14:50:39.532758 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.221.212:47889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:42215
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:39.714613 21554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:39.715098 21554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:39.715186 21554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:39.722424 21554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:39.722615 21554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:39.722842 21554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:50:39.732761 21554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.221.212:47889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:42215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:39.734891 21554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:39.736735 21554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:39.752921 21560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:39.752910 21561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:39.754190 21563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:39.754947 21554 server_base.cc:1034] running on GCE node
I20260620 14:50:39.755548 21554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:39.756564 21554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:39.757767 21554 hybrid_clock.cc:648] HybridClock initialized: now 1781967039757684 us; error 109 us; skew 500 ppm
Jun 20 14:50:39 dist-test-slave-65tz krb5kdc[20858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967039, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:39.761685 21554 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:50:39.763052 21554 webserver.cc:460] Webserver started at http://127.8.221.194:39291/ using document root <none> and password file <none>
I20260620 14:50:39.763855 21554 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:39.763952 21554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:39.764241 21554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:50:39 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967039, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
I20260620 14:50:39.781877 21554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "fe83009ecac04397becc463b8e7572ee"
format_stamp: "Formatted at 2026-06-20 14:50:39 on dist-test-slave-65tz"
server_key: "2406c0b6217d7b8df53e1c41a2c420cc"
server_key_iv: "f7d80500fe2bda9ab13d30ec56aa0772"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:39.782720 21554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "fe83009ecac04397becc463b8e7572ee"
format_stamp: "Formatted at 2026-06-20 14:50:39 on dist-test-slave-65tz"
server_key: "2406c0b6217d7b8df53e1c41a2c420cc"
server_key_iv: "f7d80500fe2bda9ab13d30ec56aa0772"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:39.795853 21554 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.005s sys 0.000s
I20260620 14:50:39.818318 21570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:39.820199 21554 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 14:50:39.820348 21554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "fe83009ecac04397becc463b8e7572ee"
format_stamp: "Formatted at 2026-06-20 14:50:39 on dist-test-slave-65tz"
server_key: "2406c0b6217d7b8df53e1c41a2c420cc"
server_key_iv: "f7d80500fe2bda9ab13d30ec56aa0772"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:39.820495 21554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:39.853420 21554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:39.858651 21554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:39.858919 21554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:39.859809 21554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:39.861104 21554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:39.861176 21554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:39.861240 21554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:39.861276 21554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:39.875211 21554 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:39115
I20260620 14:50:39.875289 21683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:39115 every 8 connection(s)
I20260620 14:50:39.876911 21554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:50:39.881666 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 21554
I20260620 14:50:39.881832 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:50:39.882167 9079 external_mini_cluster.cc:1428] Setting key 0e2cea9c0b5751a7df14366b88ee0ae6
Jun 20 14:50:39 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967039, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:50:39.898511 21551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:39.880985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:50613 (local address 127.8.221.254:42215)
0620 14:50:39.881158 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:39.881164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:39.883410 (+ 2246us) server_negotiation.cc:407] Connection header received
0620 14:50:39.884815 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:39.884818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:39.884881 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:39.884938 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:39.886860 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:39.887747 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:39.889513 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:39.889912 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:39.889986 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:39.893575 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:39.893582 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:39.893586 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:39.893633 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:39.895706 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:39.896444 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:39.896449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:39.896451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:39.896554 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:39.896976 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:39.896979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:39.896981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:39.897229 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:39.897303 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:39.898136 (+ 833us) server_negotiation.cc:299] Negotiation successful
0620 14:50:39.898311 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:50:39.900565 21686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:39.881579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:42215 (local address 127.8.221.194:50613)
0620 14:50:39.883221 (+ 1642us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:39.883284 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:39.884517 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:39.885410 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:39.885420 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:39.885892 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:39.886677 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:39.886702 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:39.887909 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:39.887913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:39.889372 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:39.889383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:39.890224 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:39.890228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:39.890456 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:39.891287 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:39.891312 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:39.893284 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:39.895878 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:39.895886 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:39.895908 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:39.896299 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:39.896665 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:39.896668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:39.896670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:39.896871 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:39.897383 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:39.897390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:39.897819 (+ 429us) client_negotiation.cc:769] Sending connection context
0620 14:50:39.898879 (+ 1060us) client_negotiation.cc:240] Negotiation successful
0620 14:50:39.899295 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1385,"thread_start_us":227,"threads_started":1}
I20260620 14:50:39.902241 21684 heartbeater.cc:347] Connected to a master server at 127.8.221.254:42215
I20260620 14:50:39.902535 21684 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.221.195@KRBTEST.COM; defaulting to no policy
I20260620 14:50:39.903211 21684 heartbeater.cc:511] Master 127.8.221.254:42215 requested a full tablet report, sending...
I20260620 14:50:39.904594 21332 ts_manager.cc:194] Registered new tserver with Master: fe83009ecac04397becc463b8e7572ee (127.8.221.194:39115)
I20260620 14:50:39.905426 21332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:50613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
I20260620 14:50:39.926963 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.221.212:47889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:42215
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:50:39 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967039, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
W20260620 14:50:40.124511 21689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:40.124974 21689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:40.125044 21689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:40.132155 21689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:40.132275 21689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:40.132375 21689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:50:40.141587 21689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.221.212:47889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:42215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:40.143492 21689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:40.145273 21689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:40.161906 21695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:40.162104 21694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:40.162309 21697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:40.163241 21689 server_base.cc:1034] running on GCE node
I20260620 14:50:40.163995 21689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:40.165098 21689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:40.168468 21689 hybrid_clock.cc:648] HybridClock initialized: now 1781967040168355 us; error 128 us; skew 500 ppm
Jun 20 14:50:40 dist-test-slave-65tz krb5kdc[20858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967040, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:40.173389 21689 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:50:40.175000 21689 webserver.cc:460] Webserver started at http://127.8.221.195:36087/ using document root <none> and password file <none>
I20260620 14:50:40.175972 21689 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:40.176123 21689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:40.176525 21689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:50:40 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967040, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
I20260620 14:50:40.212050 21689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "fa61e13020db4659bdac2bd45bc9fd4d"
format_stamp: "Formatted at 2026-06-20 14:50:40 on dist-test-slave-65tz"
server_key: "c1831c47f60637fa80c8f88d05b259e7"
server_key_iv: "63bf41bcbad577f04f759f33cb0c94c5"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:40.213050 21689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "fa61e13020db4659bdac2bd45bc9fd4d"
format_stamp: "Formatted at 2026-06-20 14:50:40 on dist-test-slave-65tz"
server_key: "c1831c47f60637fa80c8f88d05b259e7"
server_key_iv: "63bf41bcbad577f04f759f33cb0c94c5"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:40.227296 21689 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.006s sys 0.000s
I20260620 14:50:40.282099 21704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:40.284421 21689 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.004s sys 0.001s
I20260620 14:50:40.284616 21689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "fa61e13020db4659bdac2bd45bc9fd4d"
format_stamp: "Formatted at 2026-06-20 14:50:40 on dist-test-slave-65tz"
server_key: "c1831c47f60637fa80c8f88d05b259e7"
server_key_iv: "63bf41bcbad577f04f759f33cb0c94c5"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:40.284817 21689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 20 14:50:40 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967039, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
I20260620 14:50:40.339565 21689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:40.347513 21689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:40.347781 21689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:40.348618 21689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:40.349862 21689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:40.349926 21689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:40.349989 21689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:40.350019 21689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:40.382148 21689 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:33061
I20260620 14:50:40.383723 21689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:50:40.393554 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 21689
I20260620 14:50:40.393697 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:50:40.394105 9079 external_mini_cluster.cc:1428] Setting key eba9366ddc2c1dd0aae2d2a72f9873cd
I20260620 14:50:40.395097 21818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:33061 every 8 connection(s)
Jun 20 14:50:40 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967040, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:50:40.429611 21551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:40.397971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:43473 (local address 127.8.221.254:42215)
0620 14:50:40.398134 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:40.398139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:40.400425 (+ 2286us) server_negotiation.cc:407] Connection header received
0620 14:50:40.406462 (+ 6037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:40.406467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:40.406582 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:40.406653 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:40.411565 (+ 4912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:40.412452 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:40.416681 (+ 4229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:40.417073 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:40.417160 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:40.422847 (+ 5687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:40.422852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:40.422855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:40.422889 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:40.426520 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:40.427421 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:40.427426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:40.427428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:40.427525 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:40.428037 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:40.428040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:40.428042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:40.428360 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:40.428446 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:40.429252 (+ 806us) server_negotiation.cc:299] Negotiation successful
0620 14:50:40.429418 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:50:40.434706 21822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:40.398863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:42215 (local address 127.8.221.195:43473)
0620 14:50:40.400261 (+ 1398us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:40.400324 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:40.401501 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:40.409553 (+ 8052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:40.409569 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:40.410098 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:40.411370 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:40.411397 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:40.413545 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:40.413549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:40.416516 (+ 2967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:40.416531 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:40.417440 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:40.417444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:40.417672 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:40.418569 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:40.418593 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:40.422629 (+ 4036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:40.426779 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:40.426787 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:40.426816 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:40.427288 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:40.427643 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:40.427646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:40.427648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:40.427923 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:40.428514 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:40.428522 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:40.428934 (+ 412us) client_negotiation.cc:769] Sending connection context
0620 14:50:40.430194 (+ 1260us) client_negotiation.cc:240] Negotiation successful
0620 14:50:40.430639 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1132,"thread_start_us":179,"threads_started":1}
I20260620 14:50:40.436532 21820 heartbeater.cc:347] Connected to a master server at 127.8.221.254:42215
I20260620 14:50:40.436823 21820 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:40.437518 21820 heartbeater.cc:511] Master 127.8.221.254:42215 requested a full tablet report, sending...
I20260620 14:50:40.438902 21332 ts_manager.cc:194] Registered new tserver with Master: fa61e13020db4659bdac2bd45bc9fd4d (127.8.221.195:33061)
I20260620 14:50:40.439739 21332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:43473
I20260620 14:50:40.445942 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:50:40.447077 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 21415
I20260620 14:50:40.458334 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 21554
I20260620 14:50:40.467981 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 21689
I20260620 14:50:40.477531 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 21302
I20260620 14:50:40.487264 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:42215
--webserver_interface=127.8.221.254
--webserver_port=34807
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:42215
--ranger_config_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.221.212:47889/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.221.212:47889/kms with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:40.620790 21823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:40.621044 21823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:40.621081 21823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:40.625104 21823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:40.625164 21823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:40.625183 21823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:40.625200 21823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:40.625267 21823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:40.630607 21823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.221.212:47889/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:42215
--ranger_config_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:42215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=34807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:40.631667 21823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:40.632671 21823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:40.641312 21828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:40.641332 21829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:40.641353 21831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:40.641887 21823 server_base.cc:1034] running on GCE node
I20260620 14:50:40.642699 21823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:40.644392 21823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:40.645584 21823 hybrid_clock.cc:648] HybridClock initialized: now 1781967040645546 us; error 48 us; skew 500 ppm
Jun 20 14:50:40 dist-test-slave-65tz krb5kdc[20858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967040, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:40.649309 21823 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:50:40.650439 21823 webserver.cc:460] Webserver started at http://127.8.221.254:34807/ using document root <none> and password file <none>
I20260620 14:50:40.651023 21823 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:40.651103 21823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:50:40 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967040, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
I20260620 14:50:40.706015 21823 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:50:40.709561 21839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:40.711596 21823 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:50:40.711771 21823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc"
format_stamp: "Formatted at 2026-06-20 14:50:38 on dist-test-slave-65tz"
server_key: "b8db2f5fdff3ace8cfd422c3ee6d054f"
server_key_iv: "a30268be6ad094af3fa44c4db0de9e90"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:40.712680 21823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:40.730329 21823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:40.734500 21823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:40.734758 21823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:40.745775 21823 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:42215
I20260620 14:50:40.745843 21891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:42215 every 8 connection(s)
I20260620 14:50:40.747581 21823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:40.752712 21892 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:50:40.754525 21892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc: Bootstrap starting.
I20260620 14:50:40.755928 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 21823
I20260620 14:50:40.756532 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.221.212:47889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:35573
--local_ip_for_outbound_sockets=127.8.221.193
--tserver_master_addrs=127.8.221.254:42215
--webserver_port=39191
--webserver_interface=127.8.221.193
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.221.212:47889/kms with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:50:40.772415 21892 log.cc:824] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:40.780105 21892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:50:40.780596 21892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc: Bootstrap complete.
I20260620 14:50:40.786245 21892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 42215 } }
I20260620 14:50:40.786702 21892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b5c7d3d9ba24fbf960b9fa633f8f5bc, State: Initialized, Role: FOLLOWER
I20260620 14:50:40.787246 21892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 42215 } }
I20260620 14:50:40.787388 21892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:40.787451 21892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:40.787559 21892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:50:40.789559 21892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 42215 } }
I20260620 14:50:40.789964 21892 leader_election.cc:302] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b5c7d3d9ba24fbf960b9fa633f8f5bc; no voters:
I20260620 14:50:40.790351 21892 leader_election.cc:288] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:50:40.790530 21896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:50:40.790791 21896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [term 2 LEADER]: Becoming Leader. State: Replica: 1b5c7d3d9ba24fbf960b9fa633f8f5bc, State: Running, Role: LEADER
I20260620 14:50:40.791126 21896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 42215 } }
I20260620 14:50:40.791631 21892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:40.792173 21896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 42215 } } }
I20260620 14:50:40.792352 21896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:40.792624 21896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b5c7d3d9ba24fbf960b9fa633f8f5bc. Latest consensus state: current_term: 2 leader_uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b5c7d3d9ba24fbf960b9fa633f8f5bc" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 42215 } } }
I20260620 14:50:40.792768 21896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:40.794016 21902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:40.794037 21892 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:50:40.801352 21902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:40.802351 21902 catalog_manager.cc:1259] Loaded cluster ID: 55f7d27f67dd4dc6989a91152b1af781
I20260620 14:50:40.802452 21902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:40.806005 21902 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:40.809268 21902 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc: Loaded TSK: 0
I20260620 14:50:40.810796 21902 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:50:40.932581 21894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:40.933032 21894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:40.933120 21894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:40.939059 21894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:40.939175 21894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:40.939266 21894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:50:40.947453 21894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.221.212:47889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:35573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=39191
--tserver_master_addrs=127.8.221.254:42215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:40.949144 21894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:40.950647 21894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:40.964875 21932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:40.964898 21930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:40.966053 21929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:40.966632 21894 server_base.cc:1034] running on GCE node
I20260620 14:50:40.967339 21894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:40.968259 21894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:40.969493 21894 hybrid_clock.cc:648] HybridClock initialized: now 1781967040969459 us; error 51 us; skew 500 ppm
Jun 20 14:50:40 dist-test-slave-65tz krb5kdc[20858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967040, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:40.973786 21894 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:50:40.975304 21894 webserver.cc:460] Webserver started at http://127.8.221.193:39191/ using document root <none> and password file <none>
I20260620 14:50:40.976235 21894 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:40.976428 21894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:50:40 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967040, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
I20260620 14:50:41.032011 21894 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.004s sys 0.000s
I20260620 14:50:41.036188 21940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:41.038151 21894 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 14:50:41.038311 21894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "1d5b4aad460e40aeaee16e6d69808754"
format_stamp: "Formatted at 2026-06-20 14:50:39 on dist-test-slave-65tz"
server_key: "4fd6253e9bb1340584680ef4e047d403"
server_key_iv: "4b2c67c292c210496b11b23d089fe27b"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:41.039213 21894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:41.064908 21894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:41.069171 21894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:41.069429 21894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:41.070386 21894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:41.071676 21894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:41.071743 21894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:41.071805 21894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:41.071827 21894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:41.087518 21894 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:35573
I20260620 14:50:41.087541 22053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:35573 every 8 connection(s)
I20260620 14:50:41.088851 21894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:50:41.090240 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 21894
I20260620 14:50:41.090713 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.221.212:47889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:39115
--local_ip_for_outbound_sockets=127.8.221.194
--tserver_master_addrs=127.8.221.254:42215
--webserver_port=39291
--webserver_interface=127.8.221.194
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.221.212:47889/kms with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:50:41 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967040, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:50:41.117573 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.094689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:58275 (local address 127.8.221.254:42215)
0620 14:50:41.095441 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:41.095472 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:41.096100 (+ 628us) server_negotiation.cc:407] Connection header received
0620 14:50:41.098201 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:41.098227 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:41.098737 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:41.099139 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:41.101441 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.103254 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.105246 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.106028 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.106184 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.109776 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:41.109784 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:41.109841 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:41.109874 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:41.113338 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:41.114018 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:41.114025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:41.114030 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:41.114184 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:41.114561 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:41.114565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:41.114567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:41.115080 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:41.115203 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:41.115957 (+ 754us) server_negotiation.cc:299] Negotiation successful
0620 14:50:41.116331 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":144,"threads_started":1}
I20260620 14:50:41.120327 22058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.094889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:42215 (local address 127.8.221.193:58275)
0620 14:50:41.095854 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:41.095926 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:41.097542 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:41.099879 (+ 2337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:41.099893 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:41.100343 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:41.101202 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:41.101228 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.103445 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.103451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:41.105091 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:41.105103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.106540 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.106544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:41.106733 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.107521 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:41.107544 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:41.109544 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:41.113489 (+ 3945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:41.113495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:41.113511 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:41.113879 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:41.114294 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:41.114297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:41.114299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:41.114460 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:41.115288 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:41.115295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:41.115598 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 14:50:41.118715 (+ 3117us) client_negotiation.cc:240] Negotiation successful
0620 14:50:41.119116 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":204,"threads_started":1}
I20260620 14:50:41.124194 22054 heartbeater.cc:347] Connected to a master server at 127.8.221.254:42215
I20260620 14:50:41.124528 22054 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:41.125262 22054 heartbeater.cc:511] Master 127.8.221.254:42215 requested a full tablet report, sending...
W20260620 14:50:41.128201 22054 heartbeater.cc:644] Failed to heartbeat to 127.8.221.254:42215 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:50:41.128335 22054 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:41.128540 22054 heartbeater.cc:511] Master 127.8.221.254:42215 requested a full tablet report, sending...
I20260620 14:50:41.129622 22054 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:41.129817 22054 heartbeater.cc:511] Master 127.8.221.254:42215 requested a full tablet report, sending...
W20260620 14:50:41.130705 22054 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:50:41.257997 22056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:41.258399 22056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:41.258483 22056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:41.263847 22056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:41.263947 22056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:41.264051 22056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:50:41.271663 22056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.221.212:47889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:39115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=39291
--tserver_master_addrs=127.8.221.254:42215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:41.273211 22056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:41.274751 22056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:41.292184 22066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:41.292761 22063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:41.292863 22056 server_base.cc:1034] running on GCE node
W20260620 14:50:41.293591 22064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:41.294240 22056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:41.295177 22056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:41.296355 22056 hybrid_clock.cc:648] HybridClock initialized: now 1781967041296320 us; error 58 us; skew 500 ppm
Jun 20 14:50:41 dist-test-slave-65tz krb5kdc[20858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967041, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:41.300145 22056 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:50:41.301301 22056 webserver.cc:460] Webserver started at http://127.8.221.194:39291/ using document root <none> and password file <none>
I20260620 14:50:41.302093 22056 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:41.302208 22056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:50:41 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967041, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
I20260620 14:50:41.332934 22056 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:50:41.336169 22074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:41.337805 22056 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 14:50:41.338032 22056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "fe83009ecac04397becc463b8e7572ee"
format_stamp: "Formatted at 2026-06-20 14:50:39 on dist-test-slave-65tz"
server_key: "2406c0b6217d7b8df53e1c41a2c420cc"
server_key_iv: "f7d80500fe2bda9ab13d30ec56aa0772"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:41.338752 22056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:41.354559 22056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:41.358573 22056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:41.358878 22056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:41.359774 22056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:41.361043 22056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:41.361119 22056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:41.361193 22056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:41.361227 22056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:41.374146 22056 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:39115
I20260620 14:50:41.375754 22056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:50:41.376258 22188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:39115 every 8 connection(s)
I20260620 14:50:41.380491 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 22056
I20260620 14:50:41.381084 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.221.212:47889/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:33061
--local_ip_for_outbound_sockets=127.8.221.195
--tserver_master_addrs=127.8.221.254:42215
--webserver_port=36087
--webserver_interface=127.8.221.195
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.221.212:47889/kms with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:50:41 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967041, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:50:41.395819 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.379006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:38049 (local address 127.8.221.254:42215)
0620 14:50:41.379179 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:41.379183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:41.380779 (+ 1596us) server_negotiation.cc:407] Connection header received
0620 14:50:41.382516 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:41.382524 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:41.382602 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:41.382667 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:41.384503 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.385590 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.387441 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.387796 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.387882 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.390914 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:41.390918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:41.390921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:41.390953 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:41.392948 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:41.393729 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:41.393733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:41.393735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:41.393860 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:41.394297 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:41.394300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:41.394302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:41.394567 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:41.394669 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:41.395431 (+ 762us) server_negotiation.cc:299] Negotiation successful
0620 14:50:41.395601 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:50:41.397734 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.379719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:42215 (local address 127.8.221.194:38049)
0620 14:50:41.380502 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:41.380630 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:41.382207 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:41.383221 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:41.383238 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:41.383555 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:41.384306 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:41.384330 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.385789 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.385793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:41.387289 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:41.387303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.388082 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.388085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:41.388254 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.389055 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:41.389073 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:41.390702 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:41.393090 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:41.393097 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:41.393119 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:41.393594 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:41.393979 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:41.393982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:41.393984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:41.394177 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:41.394744 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:41.394751 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:41.395119 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 14:50:41.396047 (+ 928us) client_negotiation.cc:240] Negotiation successful
0620 14:50:41.396465 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":208,"threads_started":1}
I20260620 14:50:41.399859 22189 heartbeater.cc:347] Connected to a master server at 127.8.221.254:42215
I20260620 14:50:41.400216 22189 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:41.400964 22189 heartbeater.cc:511] Master 127.8.221.254:42215 requested a full tablet report, sending...
W20260620 14:50:41.402863 22189 heartbeater.cc:644] Failed to heartbeat to 127.8.221.254:42215 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:50:41.403009 22189 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:41.403218 22189 heartbeater.cc:511] Master 127.8.221.254:42215 requested a full tablet report, sending...
I20260620 14:50:41.404045 22189 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:41.404237 22189 heartbeater.cc:511] Master 127.8.221.254:42215 requested a full tablet report, sending...
W20260620 14:50:41.404954 22189 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:50:41.546634 22192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:41.547010 22192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:41.547076 22192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:41.552745 22192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:41.552843 22192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:41.552951 22192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:50:41.559162 22192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.221.212:47889/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:33061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=36087
--tserver_master_addrs=127.8.221.254:42215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:41.560317 22192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:41.561606 22192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:41.572234 22198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:41.572434 22200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:41.572732 22197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:41.573643 22192 server_base.cc:1034] running on GCE node
I20260620 14:50:41.574157 22192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:41.575009 22192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:41.576411 22192 hybrid_clock.cc:648] HybridClock initialized: now 1781967041576354 us; error 88 us; skew 500 ppm
Jun 20 14:50:41 dist-test-slave-65tz krb5kdc[20858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967041, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:41.579617 22192 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:50:41.580739 22192 webserver.cc:460] Webserver started at http://127.8.221.195:36087/ using document root <none> and password file <none>
I20260620 14:50:41.581373 22192 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:41.581487 22192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:50:41 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967041, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
I20260620 14:50:41.614454 22192 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:50:41.618412 22208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:41.620750 22192 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:50:41.620920 22192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "fa61e13020db4659bdac2bd45bc9fd4d"
format_stamp: "Formatted at 2026-06-20 14:50:40 on dist-test-slave-65tz"
server_key: "c1831c47f60637fa80c8f88d05b259e7"
server_key_iv: "63bf41bcbad577f04f759f33cb0c94c5"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:41.621884 22192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:41.639539 22192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:41.644134 22192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:41.644394 22192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:41.645357 22192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:41.646848 22192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:41.646924 22192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:41.647009 22192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:41.647051 22192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:41.659907 22192 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:33061
I20260620 14:50:41.661451 22192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:50:41.661945 22321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:33061 every 8 connection(s)
I20260620 14:50:41.666074 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 22192
Jun 20 14:50:41 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967041, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:50:41.679960 22326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.666964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33574 (local address 127.8.221.254:42215)
0620 14:50:41.669236 (+ 2272us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:41.669239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:41.669250 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:50:41.669281 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:41.669283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:41.669321 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:41.669431 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:41.670415 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.671326 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.672400 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.672656 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.672787 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.674687 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:41.674692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:41.674695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:41.674727 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:41.677592 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:41.678212 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:41.678219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:41.678230 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:41.678372 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:41.678774 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:41.678778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:41.678779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:41.679070 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:41.679336 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:41.679540 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 14:50:41.679734 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2200,"thread_start_us":154,"threads_started":1}
I20260620 14:50:41.680181 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.664398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:41827 (local address 127.8.221.254:42215)
0620 14:50:41.664636 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:41.664642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:41.665688 (+ 1046us) server_negotiation.cc:407] Connection header received
0620 14:50:41.667057 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:41.667065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:41.667141 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:41.667204 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:41.668855 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.669876 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.671361 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.671728 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.671850 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.675831 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:41.675834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:41.675836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:41.675881 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:41.677197 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:41.677990 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:41.677995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:41.677997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:41.678089 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:41.678639 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:41.678642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:41.678646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:41.678888 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:41.678967 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:41.679821 (+ 854us) server_negotiation.cc:299] Negotiation successful
0620 14:50:41.679953 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 14:50:41.682195 22324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.664866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:42215 (local address 127.8.221.195:41827)
0620 14:50:41.665453 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:41.665521 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:41.666751 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:41.667727 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:41.667735 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:41.668056 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:41.668649 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:41.668664 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.670069 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.670073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:41.671222 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:41.671233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.672001 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.672009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:41.672341 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.673187 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:41.673226 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:41.675611 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:41.677337 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:41.677345 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:41.677366 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:41.677842 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:41.678226 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:41.678229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:41.678230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:41.678496 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:41.679035 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:41.679043 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:41.679493 (+ 450us) client_negotiation.cc:769] Sending connection context
0620 14:50:41.680407 (+ 914us) client_negotiation.cc:240] Negotiation successful
0620 14:50:41.680868 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":174,"threads_started":1}
I20260620 14:50:41.683561 22322 heartbeater.cc:347] Connected to a master server at 127.8.221.254:42215
I20260620 14:50:41.683842 22322 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:41.684535 22322 heartbeater.cc:511] Master 127.8.221.254:42215 requested a full tablet report, sending...
W20260620 14:50:41.686307 22322 heartbeater.cc:644] Failed to heartbeat to 127.8.221.254:42215 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:50:41.686472 22322 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:41.686681 22322 heartbeater.cc:511] Master 127.8.221.254:42215 requested a full tablet report, sending...
I20260620 14:50:41.687613 22322 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:41.687814 22322 heartbeater.cc:511] Master 127.8.221.254:42215 requested a full tablet report, sending...
W20260620 14:50:41.688598 22322 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 14:50:42 dist-test-slave-65tz krb5kdc[20858](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967042, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:42.131001 22054 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:42.131376 22054 heartbeater.cc:511] Master 127.8.221.254:42215 requested a full tablet report, sending...
I20260620 14:50:42.133108 21856 ts_manager.cc:194] Registered new tserver with Master: 1d5b4aad460e40aeaee16e6d69808754 (127.8.221.193:35573)
I20260620 14:50:42.135097 21856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:58275
I20260620 14:50:42.405253 22189 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:42.405673 22189 heartbeater.cc:511] Master 127.8.221.254:42215 requested a full tablet report, sending...
I20260620 14:50:42.406773 21856 ts_manager.cc:194] Registered new tserver with Master: fe83009ecac04397becc463b8e7572ee (127.8.221.194:39115)
I20260620 14:50:42.407672 21856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:38049
Jun 20 14:50:42 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967042, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
I20260620 14:50:42.688874 22322 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:42.689217 22322 heartbeater.cc:511] Master 127.8.221.254:42215 requested a full tablet report, sending...
I20260620 14:50:42.690200 21856 ts_manager.cc:194] Registered new tserver with Master: fa61e13020db4659bdac2bd45bc9fd4d (127.8.221.195:33061)
I20260620 14:50:42.691001 21856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:41827
I20260620 14:50:42.695297 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:50:42.713481 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:42.699237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.8.221.254:42215)
0620 14:50:42.700211 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:42.700216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:42.700229 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:50:42.700281 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:42.700284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:42.700331 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:42.700418 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:42.701333 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.702191 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.703331 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.703674 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.703772 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:42.706315 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:42.706319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:42.706322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:42.706348 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:42.711236 (+ 4888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:42.711734 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:42.711738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:42.711740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:42.711826 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:42.712087 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:42.712090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:42.712092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:42.712306 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:42.712427 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:42.713182 (+ 755us) server_negotiation.cc:299] Negotiation successful
0620 14:50:42.713314 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":902,"thread_start_us":99,"threads_started":1}
I20260620 14:50:42.718767 21856 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:50:42.726109 21856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:50:42.763330 22351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:42.751607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:33061 (local address 127.0.0.1:47318)
0620 14:50:42.755129 (+ 3522us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:42.755141 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:42.755230 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:42.756024 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:42.756030 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:42.756053 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:42.756213 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:42.756219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.758396 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.758399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:42.759756 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:42.759767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.762708 (+ 2941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.762712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:42.762832 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:42.762872 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:50:42.763000 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 14:50:42.763152 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3467,"thread_start_us":111,"threads_started":1}
I20260620 14:50:42.763829 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:42.752080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47318 (local address 127.8.221.195:33061)
0620 14:50:42.754807 (+ 2727us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:42.754817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:42.755316 (+ 499us) server_negotiation.cc:407] Connection header received
0620 14:50:42.755407 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:42.755412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:42.755663 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:42.755809 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:42.757061 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.758266 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.759899 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.761551 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.761656 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:42.761851 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:42.763568 (+ 1717us) server_negotiation.cc:299] Negotiation successful
0620 14:50:42.763689 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2593,"thread_start_us":119,"threads_started":1}
I20260620 14:50:42.778918 22237 tablet_service.cc:1467] Processing CreateTablet for tablet 975d5a98d59c424e96ec25b097f5a06e (DEFAULT_TABLE table=test-table [id=b7eadeb0b1fa4b8eb9aed019506baaa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:42.780395 22237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975d5a98d59c424e96ec25b097f5a06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:42.780697 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:42.749533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:35573 (local address 127.0.0.1:41050)
0620 14:50:42.754285 (+ 4752us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:42.754354 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:42.754497 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:42.764068 (+ 9571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:42.764072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:42.764094 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:42.764193 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:42.764197 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.777114 (+ 12917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.777119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:42.778334 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:42.778345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.780179 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.780185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:42.780301 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:42.780338 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:50:42.780395 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 14:50:42.780507 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4532,"thread_start_us":124,"threads_started":1}
I20260620 14:50:42.783077 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:42.749770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41050 (local address 127.8.221.193:35573)
0620 14:50:42.762131 (+ 12361us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:42.762140 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:42.762171 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:50:42.762268 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:42.762274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:42.762547 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:42.764360 (+ 1813us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:42.764571 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.773180 (+ 8609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.778527 (+ 5347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.779403 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.782472 (+ 3069us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:42.782691 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:42.782757 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:50:42.782889 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12211,"thread_start_us":111,"threads_started":1}
I20260620 14:50:42.788522 21988 tablet_service.cc:1467] Processing CreateTablet for tablet 975d5a98d59c424e96ec25b097f5a06e (DEFAULT_TABLE table=test-table [id=b7eadeb0b1fa4b8eb9aed019506baaa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:42.789758 21988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975d5a98d59c424e96ec25b097f5a06e. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 14:50:42 dist-test-slave-65tz krb5kdc[20858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967042, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for HTTP/127.8.221.212@KRBTEST.COM
I20260620 14:50:42.808730 22353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:42.752834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:39115 (local address 127.0.0.1:39912)
0620 14:50:42.767192 (+ 14358us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:42.767210 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:42.767356 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:42.770585 (+ 3229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:42.770590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:42.770613 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:42.770725 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:42.770731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.775559 (+ 4828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.775564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:42.801970 (+ 26406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:42.801990 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.808256 (+ 6266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.808260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:42.808364 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:42.808390 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:50:42.808456 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:50:42.808553 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14240,"thread_start_us":128,"threads_started":1}
I20260620 14:50:42.809358 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:42.753037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39912 (local address 127.8.221.194:39115)
0620 14:50:42.754993 (+ 1956us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:42.754999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:42.769034 (+ 14035us) server_negotiation.cc:407] Connection header received
0620 14:50:42.769160 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:42.769167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:42.769469 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:42.769652 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:42.773363 (+ 3711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.775327 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.806921 (+ 31594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.807835 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.807937 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:42.808114 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:42.809094 (+ 980us) server_negotiation.cc:299] Negotiation successful
0620 14:50:42.809220 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1852,"thread_start_us":69,"threads_started":1}
I20260620 14:50:42.810182 22358 tablet_bootstrap.cc:492] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d: Bootstrap starting.
I20260620 14:50:42.812680 22358 tablet_bootstrap.cc:654] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:42.813584 22358 log.cc:824] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:42.813936 22357 tablet_bootstrap.cc:492] T 975d5a98d59c424e96ec25b097f5a06e P 1d5b4aad460e40aeaee16e6d69808754: Bootstrap starting.
I20260620 14:50:42.816682 22357 tablet_bootstrap.cc:654] T 975d5a98d59c424e96ec25b097f5a06e P 1d5b4aad460e40aeaee16e6d69808754: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:42.816545 22123 tablet_service.cc:1467] Processing CreateTablet for tablet 975d5a98d59c424e96ec25b097f5a06e (DEFAULT_TABLE table=test-table [id=b7eadeb0b1fa4b8eb9aed019506baaa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:42.817646 22357 log.cc:824] T 975d5a98d59c424e96ec25b097f5a06e P 1d5b4aad460e40aeaee16e6d69808754: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:42.818017 22123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975d5a98d59c424e96ec25b097f5a06e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:42.832588 22358 tablet_bootstrap.cc:492] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d: No bootstrap required, opened a new log
I20260620 14:50:42.832883 22358 ts_tablet_manager.cc:1397] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20260620 14:50:42.837670 22358 raft_consensus.cc:348] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5b4aad460e40aeaee16e6d69808754" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35573 } } peers { permanent_uuid: "fa61e13020db4659bdac2bd45bc9fd4d" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33061 } } peers { permanent_uuid: "fe83009ecac04397becc463b8e7572ee" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39115 } }
I20260620 14:50:42.838073 22358 raft_consensus.cc:374] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:42.838133 22358 raft_consensus.cc:729] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa61e13020db4659bdac2bd45bc9fd4d, State: Initialized, Role: FOLLOWER
I20260620 14:50:42.838682 22358 consensus_queue.cc:260] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5b4aad460e40aeaee16e6d69808754" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35573 } } peers { permanent_uuid: "fa61e13020db4659bdac2bd45bc9fd4d" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33061 } } peers { permanent_uuid: "fe83009ecac04397becc463b8e7572ee" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39115 } }
I20260620 14:50:42.840085 22358 ts_tablet_manager.cc:1428] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 14:50:42.840602 22322 heartbeater.cc:503] Master 127.8.221.254:42215 was elected leader, sending a full tablet report...
I20260620 14:50:42.845623 22357 tablet_bootstrap.cc:492] T 975d5a98d59c424e96ec25b097f5a06e P 1d5b4aad460e40aeaee16e6d69808754: No bootstrap required, opened a new log
I20260620 14:50:42.845956 22357 ts_tablet_manager.cc:1397] T 975d5a98d59c424e96ec25b097f5a06e P 1d5b4aad460e40aeaee16e6d69808754: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.000s
I20260620 14:50:42.851311 22357 raft_consensus.cc:348] T 975d5a98d59c424e96ec25b097f5a06e P 1d5b4aad460e40aeaee16e6d69808754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5b4aad460e40aeaee16e6d69808754" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35573 } } peers { permanent_uuid: "fa61e13020db4659bdac2bd45bc9fd4d" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33061 } } peers { permanent_uuid: "fe83009ecac04397becc463b8e7572ee" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39115 } }
I20260620 14:50:42.851701 22357 raft_consensus.cc:374] T 975d5a98d59c424e96ec25b097f5a06e P 1d5b4aad460e40aeaee16e6d69808754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:42.851761 22357 raft_consensus.cc:729] T 975d5a98d59c424e96ec25b097f5a06e P 1d5b4aad460e40aeaee16e6d69808754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d5b4aad460e40aeaee16e6d69808754, State: Initialized, Role: FOLLOWER
I20260620 14:50:42.852411 22357 consensus_queue.cc:260] T 975d5a98d59c424e96ec25b097f5a06e P 1d5b4aad460e40aeaee16e6d69808754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5b4aad460e40aeaee16e6d69808754" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35573 } } peers { permanent_uuid: "fa61e13020db4659bdac2bd45bc9fd4d" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33061 } } peers { permanent_uuid: "fe83009ecac04397becc463b8e7572ee" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39115 } }
I20260620 14:50:42.853703 22357 ts_tablet_manager.cc:1428] T 975d5a98d59c424e96ec25b097f5a06e P 1d5b4aad460e40aeaee16e6d69808754: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260620 14:50:42.854221 22054 heartbeater.cc:503] Master 127.8.221.254:42215 was elected leader, sending a full tablet report...
I20260620 14:50:42.856674 22361 tablet_bootstrap.cc:492] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee: Bootstrap starting.
I20260620 14:50:42.859782 22361 tablet_bootstrap.cc:654] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:42.860843 22361 log.cc:824] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:42.863790 22361 tablet_bootstrap.cc:492] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee: No bootstrap required, opened a new log
I20260620 14:50:42.864032 22361 ts_tablet_manager.cc:1397] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260620 14:50:42.868643 22361 raft_consensus.cc:348] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5b4aad460e40aeaee16e6d69808754" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35573 } } peers { permanent_uuid: "fa61e13020db4659bdac2bd45bc9fd4d" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33061 } } peers { permanent_uuid: "fe83009ecac04397becc463b8e7572ee" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39115 } }
I20260620 14:50:42.869005 22361 raft_consensus.cc:374] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:42.869057 22361 raft_consensus.cc:729] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe83009ecac04397becc463b8e7572ee, State: Initialized, Role: FOLLOWER
I20260620 14:50:42.869656 22361 consensus_queue.cc:260] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5b4aad460e40aeaee16e6d69808754" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35573 } } peers { permanent_uuid: "fa61e13020db4659bdac2bd45bc9fd4d" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33061 } } peers { permanent_uuid: "fe83009ecac04397becc463b8e7572ee" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39115 } }
I20260620 14:50:42.870958 22361 ts_tablet_manager.cc:1428] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee: Time spent starting tablet: real 0.007s user 0.000s sys 0.008s
I20260620 14:50:42.871413 22189 heartbeater.cc:503] Master 127.8.221.254:42215 was elected leader, sending a full tablet report...
W20260620 14:50:42.878492 22190 tablet.cc:2369] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:50:42.897042 22365 raft_consensus.cc:482] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:50:42.897370 22365 raft_consensus.cc:504] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5b4aad460e40aeaee16e6d69808754" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35573 } } peers { permanent_uuid: "fa61e13020db4659bdac2bd45bc9fd4d" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33061 } } peers { permanent_uuid: "fe83009ecac04397becc463b8e7572ee" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39115 } }
I20260620 14:50:42.899214 22365 leader_election.cc:288] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d5b4aad460e40aeaee16e6d69808754 (127.8.221.193:35573), fa61e13020db4659bdac2bd45bc9fd4d (127.8.221.195:33061)
W20260620 14:50:42.915104 22323 tablet.cc:2369] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:50:42.916397 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:42.900115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:38587 (local address 127.8.221.193:35573)
0620 14:50:42.900275 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:42.900281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:42.905093 (+ 4812us) server_negotiation.cc:407] Connection header received
0620 14:50:42.908577 (+ 3484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:42.908581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:42.908645 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:42.908697 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:42.909202 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.910011 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.915017 (+ 5006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.915608 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.915715 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:42.915778 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:42.916129 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 14:50:42.916226 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:50:42.916978 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:42.899556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:35573 (local address 127.8.221.194:38587)
0620 14:50:42.904965 (+ 5409us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:42.904984 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:42.908381 (+ 3397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:42.908866 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:42.908869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:42.908890 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:42.909066 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:42.909074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.910162 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.910165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:42.911566 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:42.911578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.915877 (+ 4299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.915880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:42.915974 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:42.915998 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:50:42.916696 (+ 698us) client_negotiation.cc:240] Negotiation successful
0620 14:50:42.916791 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5249,"thread_start_us":185,"threads_started":1}
I20260620 14:50:42.917898 22003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975d5a98d59c424e96ec25b097f5a06e" candidate_uuid: "fe83009ecac04397becc463b8e7572ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5b4aad460e40aeaee16e6d69808754" is_pre_election: true
I20260620 14:50:42.918358 22003 raft_consensus.cc:2454] T 975d5a98d59c424e96ec25b097f5a06e P 1d5b4aad460e40aeaee16e6d69808754 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe83009ecac04397becc463b8e7572ee in term 0.
I20260620 14:50:42.919162 22076 leader_election.cc:302] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d5b4aad460e40aeaee16e6d69808754, fe83009ecac04397becc463b8e7572ee; no voters:
I20260620 14:50:42.919703 22365 raft_consensus.cc:2781] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:50:42.919811 22365 raft_consensus.cc:482] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:50:42.919863 22365 raft_consensus.cc:3037] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:42.921432 22365 raft_consensus.cc:504] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5b4aad460e40aeaee16e6d69808754" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35573 } } peers { permanent_uuid: "fa61e13020db4659bdac2bd45bc9fd4d" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33061 } } peers { permanent_uuid: "fe83009ecac04397becc463b8e7572ee" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39115 } }
I20260620 14:50:42.921989 22365 leader_election.cc:288] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [CANDIDATE]: Term 1 election: Requested vote from peers 1d5b4aad460e40aeaee16e6d69808754 (127.8.221.193:35573), fa61e13020db4659bdac2bd45bc9fd4d (127.8.221.195:33061)
I20260620 14:50:42.922609 22003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975d5a98d59c424e96ec25b097f5a06e" candidate_uuid: "fe83009ecac04397becc463b8e7572ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5b4aad460e40aeaee16e6d69808754"
I20260620 14:50:42.922770 22003 raft_consensus.cc:3037] T 975d5a98d59c424e96ec25b097f5a06e P 1d5b4aad460e40aeaee16e6d69808754 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:42.924154 22003 raft_consensus.cc:2454] T 975d5a98d59c424e96ec25b097f5a06e P 1d5b4aad460e40aeaee16e6d69808754 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe83009ecac04397becc463b8e7572ee in term 1.
I20260620 14:50:42.924523 22076 leader_election.cc:302] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d5b4aad460e40aeaee16e6d69808754, fe83009ecac04397becc463b8e7572ee; no voters:
I20260620 14:50:42.924718 22365 raft_consensus.cc:2781] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:42.924970 22365 raft_consensus.cc:686] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [term 1 LEADER]: Becoming Leader. State: Replica: fe83009ecac04397becc463b8e7572ee, State: Running, Role: LEADER
I20260620 14:50:42.925423 22365 consensus_queue.cc:237] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5b4aad460e40aeaee16e6d69808754" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35573 } } peers { permanent_uuid: "fa61e13020db4659bdac2bd45bc9fd4d" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33061 } } peers { permanent_uuid: "fe83009ecac04397becc463b8e7572ee" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39115 } }
I20260620 14:50:42.929730 21856 catalog_manager.cc:5495] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee reported cstate change: term changed from 0 to 1, leader changed from <none> to fe83009ecac04397becc463b8e7572ee (127.8.221.194). New cstate: current_term: 1 leader_uuid: "fe83009ecac04397becc463b8e7572ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d5b4aad460e40aeaee16e6d69808754" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa61e13020db4659bdac2bd45bc9fd4d" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe83009ecac04397becc463b8e7572ee" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39115 } health_report { overall_health: HEALTHY } } }
I20260620 14:50:42.934679 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:42.900139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:57691 (local address 127.8.221.195:33061)
0620 14:50:42.900378 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:42.900383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:42.911938 (+ 11555us) server_negotiation.cc:407] Connection header received
0620 14:50:42.912133 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:42.912137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:42.912194 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:42.912240 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:42.912625 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.913343 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.932512 (+ 19169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.933235 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.933364 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:42.933441 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:42.934402 (+ 961us) server_negotiation.cc:299] Negotiation successful
0620 14:50:42.934519 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260620 14:50:42.935191 22366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:42.899597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:33061 (local address 127.8.221.194:57691)
0620 14:50:42.911836 (+ 12239us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:42.911852 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:42.912015 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:42.912376 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:42.912379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:42.912393 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:42.912515 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:42.912520 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.913497 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.913500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:42.914858 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:42.914868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.933573 (+ 18705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.933578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:42.934236 (+ 658us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:42.934269 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:50:42.934916 (+ 647us) client_negotiation.cc:240] Negotiation successful
0620 14:50:42.935029 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12156,"thread_start_us":155,"threads_started":1}
I20260620 14:50:42.936036 22276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975d5a98d59c424e96ec25b097f5a06e" candidate_uuid: "fe83009ecac04397becc463b8e7572ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa61e13020db4659bdac2bd45bc9fd4d" is_pre_election: true
I20260620 14:50:42.936504 22276 raft_consensus.cc:2454] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe83009ecac04397becc463b8e7572ee in term 0.
I20260620 14:50:42.936980 22275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975d5a98d59c424e96ec25b097f5a06e" candidate_uuid: "fe83009ecac04397becc463b8e7572ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa61e13020db4659bdac2bd45bc9fd4d"
I20260620 14:50:42.937131 22275 raft_consensus.cc:3037] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:42.939231 22275 raft_consensus.cc:2454] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe83009ecac04397becc463b8e7572ee in term 1.
I20260620 14:50:42.958264 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:42.952587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39924 (local address 127.8.221.194:39115)
0620 14:50:42.952754 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:42.952759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:42.952784 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:50:42.952915 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:42.952919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:42.953006 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:42.953140 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:42.953570 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.954429 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.955707 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:42.956053 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:42.956150 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:42.956617 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:42.957338 (+ 721us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:42.957472 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:42.957983 (+ 511us) server_negotiation.cc:299] Negotiation successful
0620 14:50:42.958058 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:50:42.967768 22275 raft_consensus.cc:1261] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d [term 1 FOLLOWER]: Refusing update from remote peer fe83009ecac04397becc463b8e7572ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:42.967926 22003 raft_consensus.cc:1261] T 975d5a98d59c424e96ec25b097f5a06e P 1d5b4aad460e40aeaee16e6d69808754 [term 1 FOLLOWER]: Refusing update from remote peer fe83009ecac04397becc463b8e7572ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:42.968919 22368 consensus_queue.cc:1035] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d5b4aad460e40aeaee16e6d69808754" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 35573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:42.969659 22368 consensus_queue.cc:1035] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa61e13020db4659bdac2bd45bc9fd4d" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:42.979498 22371 mvcc.cc:205] Tried to move back new op lower bound from 7298937007982465024 to 7298937007826444288. Current Snapshot: MvccSnapshot[applied={T|T < 7298937007982465024}]
I20260620 14:50:42.982779 22373 mvcc.cc:205] Tried to move back new op lower bound from 7298937007982465024 to 7298937007826444288. Current Snapshot: MvccSnapshot[applied={T|T < 7298937007982465024}]
I20260620 14:50:42.997231 22372 mvcc.cc:205] Tried to move back new op lower bound from 7298937007982465024 to 7298937007826444288. Current Snapshot: MvccSnapshot[applied={T|T < 7298937007982465024}]
I20260620 14:50:43.002756 21855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:43.002974 21855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:43.008687 21855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b5c7d3d9ba24fbf960b9fa633f8f5bc: Sending DeleteTablet for 3 replicas of tablet 975d5a98d59c424e96ec25b097f5a06e
I20260620 14:50:43.010270 21988 tablet_service.cc:1514] Processing DeleteTablet for tablet 975d5a98d59c424e96ec25b097f5a06e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:43 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:41050
I20260620 14:50:43.010926 22123 tablet_service.cc:1514] Processing DeleteTablet for tablet 975d5a98d59c424e96ec25b097f5a06e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:43 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:39912
I20260620 14:50:43.012049 22237 tablet_service.cc:1514] Processing DeleteTablet for tablet 975d5a98d59c424e96ec25b097f5a06e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:43 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:47318
I20260620 14:50:43.013535 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 21894
I20260620 14:50:43.021369 22382 tablet_replica.cc:330] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d: stopping tablet replica
I20260620 14:50:43.021718 22381 tablet_replica.cc:330] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee: stopping tablet replica
I20260620 14:50:43.023258 22381 raft_consensus.cc:2229] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:50:43.023860 22381 raft_consensus.cc:2258] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:43.024453 22382 raft_consensus.cc:2229] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:43.024914 22382 raft_consensus.cc:2258] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:43.027058 22382 ts_tablet_manager.cc:1905] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:43.029928 22381 ts_tablet_manager.cc:1905] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:43.033172 22381 ts_tablet_manager.cc:1918] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:43.033291 22381 log.cc:1196] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/975d5a98d59c424e96ec25b097f5a06e
I20260620 14:50:43.033669 22381 ts_tablet_manager.cc:1939] T 975d5a98d59c424e96ec25b097f5a06e P fe83009ecac04397becc463b8e7572ee: Deleting consensus metadata
I20260620 14:50:43.036893 22382 ts_tablet_manager.cc:1918] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:43.037024 22382 log.cc:1196] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/975d5a98d59c424e96ec25b097f5a06e
I20260620 14:50:43.037405 22382 ts_tablet_manager.cc:1939] T 975d5a98d59c424e96ec25b097f5a06e P fa61e13020db4659bdac2bd45bc9fd4d: Deleting consensus metadata
I20260620 14:50:43.043231 21842 catalog_manager.cc:4841] TS fa61e13020db4659bdac2bd45bc9fd4d (127.8.221.195:33061): tablet 975d5a98d59c424e96ec25b097f5a06e (table test-table [id=b7eadeb0b1fa4b8eb9aed019506baaa0]) successfully deleted
I20260620 14:50:43.045104 21841 catalog_manager.cc:4841] TS fe83009ecac04397becc463b8e7572ee (127.8.221.194:39115): tablet 975d5a98d59c424e96ec25b097f5a06e (table test-table [id=b7eadeb0b1fa4b8eb9aed019506baaa0]) successfully deleted
W20260620 14:50:43.045431 21841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.221.193:35573 (error 108)
I20260620 14:50:43.045744 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 22056
I20260620 14:50:43.053011 22353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:43.052558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:35573 (local address 127.0.0.1:41062)
0620 14:50:43.052799 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:43.052898 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.193:35573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260620 14:50:43.053233 21841 catalog_manager.cc:4584] TS 1d5b4aad460e40aeaee16e6d69808754 (127.8.221.193:35573): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 975d5a98d59c424e96ec25b097f5a06e: Network error: Client connection negotiation failed: client connection to 127.8.221.193:35573: connect: Connection refused (error 111)
I20260620 14:50:43.059017 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 22192
I20260620 14:50:43.068562 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 21823
I20260620 14:50:43.078490 9079 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:50:43.414400 9079 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:50:43.414592 9079 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:50:43.760200 9079 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:50:43.760 UTC [20888] LOG: received smart shutdown request
2026-06-20 14:50:43.764 UTC [20888] LOG: background worker "logical replication launcher" (PID 20897) exited with exit code 1
2026-06-20 14:50:43.765 UTC [20892] LOG: shutting down
2026-06-20 14:50:43.942 UTC [20888] LOG: database system is shut down
2026-06-20T14:50:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52460 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:44 dist-test-slave-65tz krb5kdc[22392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:44 dist-test-slave-65tz krb5kdc[22392](info): set up 2 sockets
Jun 20 14:50:44 dist-test-slave-65tz krb5kdc[22392](info): commencing operation
krb5kdc: starting...
W20260620 14:50:45.130304 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:50:45 dist-test-slave-65tz krb5kdc[22392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:50:45.258762 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38395
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:38395,127.8.221.253:45249,127.8.221.252:46229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:45.371287 22405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:45.371562 22405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:45.371604 22405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:45.375619 22405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:45.375689 22405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:45.375711 22405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:45.375728 22405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:45.375744 22405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:45.381093 22405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:38395,127.8.221.253:45249,127.8.221.252:46229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:45.382143 22405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:45.383114 22405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:45.392781 22411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:45.392781 22410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:45.392861 22413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:45.393448 22405 server_base.cc:1034] running on GCE node
I20260620 14:50:45.394323 22405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:45.395788 22405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:45.397008 22405 hybrid_clock.cc:648] HybridClock initialized: now 1781967045396974 us; error 56 us; skew 500 ppm
Jun 20 14:50:45 dist-test-slave-65tz krb5kdc[22392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967045, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:45.400609 22405 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:50:45.401736 22405 webserver.cc:460] Webserver started at http://127.8.221.254:46305/ using document root <none> and password file <none>
I20260620 14:50:45.402468 22405 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:45.402582 22405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:45.402870 22405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:45.405380 22405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "83b44b03c3844be0bc949cf620199fc2"
format_stamp: "Formatted at 2026-06-20 14:50:45 on dist-test-slave-65tz"
server_key: "8a24677527310feaecd92095f869d70a"
server_key_iv: "7dc224c72ca7974b2ce927c5b12e8946"
server_key_version: "encryptionkey@0"
I20260620 14:50:45.406101 22405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "83b44b03c3844be0bc949cf620199fc2"
format_stamp: "Formatted at 2026-06-20 14:50:45 on dist-test-slave-65tz"
server_key: "8a24677527310feaecd92095f869d70a"
server_key_iv: "7dc224c72ca7974b2ce927c5b12e8946"
server_key_version: "encryptionkey@0"
I20260620 14:50:45.410900 22405 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:50:45.413697 22420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:45.415278 22405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:50:45.415467 22405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "83b44b03c3844be0bc949cf620199fc2"
format_stamp: "Formatted at 2026-06-20 14:50:45 on dist-test-slave-65tz"
server_key: "8a24677527310feaecd92095f869d70a"
server_key_iv: "7dc224c72ca7974b2ce927c5b12e8946"
server_key_version: "encryptionkey@0"
I20260620 14:50:45.415609 22405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:45.432349 22405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:45.435729 22405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:45.435941 22405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:45.443817 22405 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:38395
I20260620 14:50:45.443893 22472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:38395 every 8 connection(s)
I20260620 14:50:45.444808 22405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:45.445834 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 22405
I20260620 14:50:45.446013 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:50:45.446419 9079 external_mini_cluster.cc:1428] Setting key a00e4d5f0d1b25c0c6f30abfd243fd20
I20260620 14:50:45.449177 22473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:45.453989 22473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.221.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.253@KRBTEST.COM" created.
Jun 20 14:50:45 dist-test-slave-65tz krb5kdc[22392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967045, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:50:45.471411 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.456957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38395 (local address 127.0.0.1:60912)
0620 14:50:45.458164 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.458209 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:45.459189 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:45.459915 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:45.459921 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:45.460382 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:45.460761 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:45.460781 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.462165 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.462169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:45.462932 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:45.462943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.463585 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.463589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:45.463694 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:45.464313 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:45.464330 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:45.466172 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:45.468258 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:45.468266 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:45.468275 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:45.468557 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:45.468872 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:45.468875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:45.468876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:45.469021 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:45.469666 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:45.469672 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:45.469858 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 14:50:45.470046 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 14:50:45.470392 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"spinlock_wait_cycles":13056,"thread_start_us":202,"threads_started":1}
I20260620 14:50:45.471419 22476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.456957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60912 (local address 127.8.221.254:38395)
0620 14:50:45.457597 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:45.457621 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:45.458375 (+ 754us) server_negotiation.cc:407] Connection header received
0620 14:50:45.459520 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:45.459524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:45.459718 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:45.459924 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:45.460990 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.462002 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.463062 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.463433 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.463604 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:45.466319 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:45.466332 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:45.466341 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:45.466361 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:45.468103 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:45.468663 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:45.468669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:45.468670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:45.468773 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:45.469107 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:45.469110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:45.469111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:45.469502 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:45.469684 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:45.470162 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 14:50:45.470450 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":441,"spinlock_wait_cycles":1280,"thread_start_us":202,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.253.keytab.
Entry for principal kudu/127.8.221.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.253.keytab.
I20260620 14:50:45.473273 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.253:45249
--webserver_interface=127.8.221.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.253.keytab
--principal=kudu/127.8.221.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:38395,127.8.221.253:45249,127.8.221.252:46229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:50:45.474411 22473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:45.475274 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.475024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:45249 (local address 127.0.0.1:33178)
0620 14:50:45.475142 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.475216 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:45249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 14:50:45.475558 22421 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.221.253:45249: connect: Connection refused (error 111)
I20260620 14:50:45.476670 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.476398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:45249 (local address 127.0.0.1:33192)
0620 14:50:45.476571 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.476620 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:45249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260620 14:50:45.476917 22473 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.253:45249: Network error: Client connection negotiation failed: client connection to 127.8.221.253:45249: connect: Connection refused (error 111)
I20260620 14:50:45.526172 22473 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } attempt: 1
I20260620 14:50:45.527071 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.526779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:45249 (local address 127.0.0.1:33196)
0620 14:50:45.526931 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.526999 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:45249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 14:50:45.527832 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.527627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:45249 (local address 127.0.0.1:33202)
0620 14:50:45.527731 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.527784 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:45249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 14:50:45.528014 22473 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.253:45249: Network error: Client connection negotiation failed: client connection to 127.8.221.253:45249: connect: Connection refused (error 111)
I20260620 14:50:45.596179 22473 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } attempt: 2
I20260620 14:50:45.597144 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.596827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:45249 (local address 127.0.0.1:33206)
0620 14:50:45.596967 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.597035 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:45249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 14:50:45.598146 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.597629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:45249 (local address 127.0.0.1:33220)
0620 14:50:45.598013 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.598070 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:45249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":119,"threads_started":1}
W20260620 14:50:45.598266 22473 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.253:45249: Network error: Client connection negotiation failed: client connection to 127.8.221.253:45249: connect: Connection refused (error 111)
W20260620 14:50:45.602710 22479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:45.603000 22479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:45.603085 22479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:45.607373 22479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:45.607445 22479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:45.607467 22479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:45.607486 22479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:45.607545 22479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:45.613472 22479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:38395,127.8.221.253:45249,127.8.221.252:46229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.253.keytab
--principal=kudu/127.8.221.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.253:45249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.221.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:45.614832 22479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:45.616062 22479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:45.625929 22489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:45.625937 22487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:45.625955 22486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:45.626204 22479 server_base.cc:1034] running on GCE node
I20260620 14:50:45.626883 22479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:45.628147 22479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:45.629340 22479 hybrid_clock.cc:648] HybridClock initialized: now 1781967045629327 us; error 32 us; skew 500 ppm
Jun 20 14:50:45 dist-test-slave-65tz krb5kdc[22392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967045, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:45.632484 22479 init.cc:377] Logged in from keytab as kudu/127.8.221.253@KRBTEST.COM (short username kudu)
I20260620 14:50:45.633437 22479 webserver.cc:460] Webserver started at http://127.8.221.253:37593/ using document root <none> and password file <none>
I20260620 14:50:45.634111 22479 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:45.634179 22479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:45.634403 22479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:45.636484 22479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-1/data/instance:
uuid: "3aded7d26e9f425685994de6ecba0302"
format_stamp: "Formatted at 2026-06-20 14:50:45 on dist-test-slave-65tz"
server_key: "f6454d55e23d44de81ba7d2b1e3f7076"
server_key_iv: "53b16b38684d9e4ad0fe041da0fbf3ee"
server_key_version: "encryptionkey@0"
I20260620 14:50:45.637058 22479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-1/wal/instance:
uuid: "3aded7d26e9f425685994de6ecba0302"
format_stamp: "Formatted at 2026-06-20 14:50:45 on dist-test-slave-65tz"
server_key: "f6454d55e23d44de81ba7d2b1e3f7076"
server_key_iv: "53b16b38684d9e4ad0fe041da0fbf3ee"
server_key_version: "encryptionkey@0"
I20260620 14:50:45.641068 22479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:50:45.644444 22496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:45.646034 22479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:45.646188 22479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-1/wal
uuid: "3aded7d26e9f425685994de6ecba0302"
format_stamp: "Formatted at 2026-06-20 14:50:45 on dist-test-slave-65tz"
server_key: "f6454d55e23d44de81ba7d2b1e3f7076"
server_key_iv: "53b16b38684d9e4ad0fe041da0fbf3ee"
server_key_version: "encryptionkey@0"
I20260620 14:50:45.646309 22479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:45.669334 22479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:45.672974 22479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:45.673202 22479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:45.681403 22479 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.253:45249
I20260620 14:50:45.681432 22548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.253:45249 every 8 connection(s)
I20260620 14:50:45.682559 22479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-1/data/info.pb
I20260620 14:50:45.686291 22549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:45.689431 22473 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } attempt: 3
I20260620 14:50:45.691293 22549 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:45.692492 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 22479
I20260620 14:50:45.692653 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-1/wal/instance
I20260620 14:50:45.692991 9079 external_mini_cluster.cc:1428] Setting key dc6f677fc8176ef4ab90570134155a5c
Jun 20 14:50:45 dist-test-slave-65tz krb5kdc[22392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967045, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for kudu/127.8.221.253@KRBTEST.COM
Jun 20 14:50:45 dist-test-slave-65tz krb5kdc[22392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967045, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.253@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:50:45.704407 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.690063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:45249 (local address 127.0.0.1:33232)
0620 14:50:45.690310 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.690335 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:45.690504 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:45.692966 (+ 2462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:45.692969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:45.693207 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:45.693343 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:45.693350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.695007 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.695012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:45.695853 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:45.695863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.696745 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.696748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:45.696836 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:45.697424 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:45.697445 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:45.699297 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:45.702154 (+ 2857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:45.702158 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:45.702160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:45.702514 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:45.702893 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:45.702897 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:45.702899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:45.702997 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:45.703695 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:45.703699 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:45.703815 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 14:50:45.703946 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 14:50:45.704181 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"spinlock_wait_cycles":8064}
WARNING: no policy specified for kudu/127.8.221.252@KRBTEST.COM; defaulting to no policy
I20260620 14:50:45.705660 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.690812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33232 (local address 127.8.221.253:45249)
0620 14:50:45.691278 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:45.691294 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:45.691333 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:50:45.692143 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:45.692160 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:45.692613 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:45.693026 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:45.693520 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.694839 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.696030 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.696596 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.696762 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:45.699515 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:45.699524 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:45.699540 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:45.699586 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:45.702003 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:45.702642 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:45.702649 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:45.702655 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:45.702789 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:45.703098 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:45.703102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:45.703103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:45.703578 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:45.703734 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:45.704111 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 14:50:45.704460 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":159,"threads_started":1}
I20260620 14:50:45.706893 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.694371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38395 (local address 127.0.0.1:60916)
0620 14:50:45.694839 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.694890 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:45.695053 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:45.695403 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:45.695408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:45.695818 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:45.695994 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:45.696002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.697752 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.697755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:45.698521 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:45.698531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.699210 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.699215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:45.699315 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:45.699864 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:45.699886 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:45.701982 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:45.704390 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:45.704403 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:45.704410 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:45.705118 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:45.705564 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:45.705569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:45.705572 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:45.705686 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:45.706230 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:45.706235 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:45.706344 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 14:50:45.706554 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 14:50:45.706761 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":162,"threads_started":1}
I20260620 14:50:45.706979 22476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.694342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60916 (local address 127.8.221.254:38395)
0620 14:50:45.694449 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:45.694453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:45.694957 (+ 504us) server_negotiation.cc:407] Connection header received
0620 14:50:45.695156 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:45.695159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:45.695199 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:45.695247 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:45.696158 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.697593 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.698649 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.699018 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.699132 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:45.702151 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:45.702155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:45.702158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:45.702211 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:45.704239 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:45.705282 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:45.705286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:45.705288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:45.705383 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:45.705847 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:45.705852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:45.705856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:45.706107 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:45.706238 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:45.706639 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 14:50:45.706774 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42,"spinlock_wait_cycles":16256}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.252@KRBTEST.COM" created.
I20260620 14:50:45.708276 22549 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:45.708550 22473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:45.709317 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.709008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:46229 (local address 127.0.0.1:33450)
0620 14:50:45.709181 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.709253 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 14:50:45.710492 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.710215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:46229 (local address 127.0.0.1:33466)
0620 14:50:45.710389 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.710443 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260620 14:50:45.710723 22473 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.252:46229: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
Jun 20 14:50:45 dist-test-slave-65tz krb5kdc[22392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967045, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.253@KRBTEST.COM for kudu/127.8.221.253@KRBTEST.COM
I20260620 14:50:45.719007 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.708907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:45249 (local address 127.0.0.1:33234)
0620 14:50:45.709007 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.709023 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:45.709103 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:45.709709 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:45.709712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:45.709982 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:45.710099 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:45.710110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.711236 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.711243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:45.712217 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:45.712231 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.712755 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.712761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:45.712891 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:45.713494 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:45.713515 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:45.715170 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:45.717081 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:45.717087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:45.717090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:45.717300 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:45.717610 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:45.717615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:45.717617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:45.717705 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:45.718143 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:45.718147 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:45.718241 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 14:50:45.718409 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 14:50:45.718629 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 14:50:45.719007 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.709048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33234 (local address 127.8.221.253:45249)
0620 14:50:45.709527 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:45.709530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:45.709541 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:50:45.709568 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:45.709571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:45.709602 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:45.709702 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:45.710245 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.711105 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.712365 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.712620 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.712723 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:45.715329 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:45.715334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:45.715337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:45.715369 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:45.716947 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:45.717424 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:45.717429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:45.717431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:45.717512 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:45.717834 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:45.717837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:45.717839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:45.718035 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:45.718157 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:45.718457 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 14:50:45.718697 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":410}
I20260620 14:50:45.720189 22549 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:45.721190 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.720848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:46229 (local address 127.0.0.1:33474)
0620 14:50:45.721043 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.721136 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260620 14:50:45.721885 22498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
I20260620 14:50:45.722901 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.722690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:46229 (local address 127.0.0.1:33482)
0620 14:50:45.722796 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.722843 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 14:50:45.723083 22549 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.252:46229: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.252.keytab.
Entry for principal kudu/127.8.221.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.252.keytab.
I20260620 14:50:45.725431 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.252:46229
--webserver_interface=127.8.221.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.252.keytab
--principal=kudu/127.8.221.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:38395,127.8.221.253:45249,127.8.221.252:46229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:50:45.741889 22473 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } attempt: 1
I20260620 14:50:45.743088 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.742698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:46229 (local address 127.0.0.1:33494)
0620 14:50:45.742915 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.743004 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 14:50:45.743886 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.743681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:46229 (local address 127.0.0.1:33500)
0620 14:50:45.743780 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.743834 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 14:50:45.744025 22473 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.252:46229: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
I20260620 14:50:45.772359 22549 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } attempt: 1
I20260620 14:50:45.773533 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.773149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:46229 (local address 127.0.0.1:33514)
0620 14:50:45.773356 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.773449 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 14:50:45.774549 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.774287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:46229 (local address 127.0.0.1:33520)
0620 14:50:45.774411 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.774498 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 14:50:45.774688 22549 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.252:46229: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
I20260620 14:50:45.819202 22473 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } attempt: 2
I20260620 14:50:45.820458 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.820021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:46229 (local address 127.0.0.1:33530)
0620 14:50:45.820217 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.820331 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 14:50:45.821264 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.821047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:46229 (local address 127.0.0.1:33542)
0620 14:50:45.821152 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.821212 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 14:50:45.821483 22473 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.252:46229: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
I20260620 14:50:45.842882 22549 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } attempt: 2
I20260620 14:50:45.843713 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.843438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:46229 (local address 127.0.0.1:33550)
0620 14:50:45.843575 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.843641 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 14:50:45.844390 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.844187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:46229 (local address 127.0.0.1:33554)
0620 14:50:45.844295 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.844345 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 14:50:45.844543 22549 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.252:46229: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
W20260620 14:50:45.861425 22556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:45.861804 22556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:45.861927 22556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:45.866269 22556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:45.866359 22556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:45.866396 22556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:45.866456 22556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:45.866513 22556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:45.872392 22556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:38395,127.8.221.253:45249,127.8.221.252:46229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.252.keytab
--principal=kudu/127.8.221.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.252:46229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.221.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:45.873932 22556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:45.875267 22556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:45.885468 22562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:45.885464 22561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:45.885494 22564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:45.885914 22556 server_base.cc:1034] running on GCE node
I20260620 14:50:45.886624 22556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:45.887984 22556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:45.889262 22556 hybrid_clock.cc:648] HybridClock initialized: now 1781967045889173 us; error 60 us; skew 500 ppm
Jun 20 14:50:45 dist-test-slave-65tz krb5kdc[22392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967045, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:45.892887 22556 init.cc:377] Logged in from keytab as kudu/127.8.221.252@KRBTEST.COM (short username kudu)
I20260620 14:50:45.894140 22556 webserver.cc:460] Webserver started at http://127.8.221.252:33307/ using document root <none> and password file <none>
I20260620 14:50:45.894901 22556 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:45.895006 22556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:45.895309 22556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:45.898063 22556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-2/data/instance:
uuid: "aca4f423e004496ebc7a8760d5d66295"
format_stamp: "Formatted at 2026-06-20 14:50:45 on dist-test-slave-65tz"
server_key: "13035d9896bac7d62b37716d2e8cac9b"
server_key_iv: "a07410d140735eaa01c520b19c36cab8"
server_key_version: "encryptionkey@0"
I20260620 14:50:45.898732 22556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-2/wal/instance:
uuid: "aca4f423e004496ebc7a8760d5d66295"
format_stamp: "Formatted at 2026-06-20 14:50:45 on dist-test-slave-65tz"
server_key: "13035d9896bac7d62b37716d2e8cac9b"
server_key_iv: "a07410d140735eaa01c520b19c36cab8"
server_key_version: "encryptionkey@0"
I20260620 14:50:45.903409 22556 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 14:50:45.906569 22571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:45.908201 22556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:50:45.908385 22556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-2/wal
uuid: "aca4f423e004496ebc7a8760d5d66295"
format_stamp: "Formatted at 2026-06-20 14:50:45 on dist-test-slave-65tz"
server_key: "13035d9896bac7d62b37716d2e8cac9b"
server_key_iv: "a07410d140735eaa01c520b19c36cab8"
server_key_version: "encryptionkey@0"
I20260620 14:50:45.908519 22556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:45.920660 22473 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } attempt: 3
I20260620 14:50:45.921567 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.921241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:46229 (local address 127.0.0.1:33566)
0620 14:50:45.921412 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.921488 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 14:50:45.922319 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.922123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:46229 (local address 127.0.0.1:33574)
0620 14:50:45.922219 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.922273 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 14:50:45.922433 22473 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.252:46229: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
I20260620 14:50:45.935720 22549 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } attempt: 3
I20260620 14:50:45.936776 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.936436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:46229 (local address 127.0.0.1:33586)
0620 14:50:45.936620 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.936697 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 14:50:45.937610 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.937337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:46229 (local address 127.0.0.1:33590)
0620 14:50:45.937502 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.937560 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260620 14:50:45.937739 22549 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.252:46229: Network error: Client connection negotiation failed: client connection to 127.8.221.252:46229: connect: Connection refused (error 111)
I20260620 14:50:45.950205 22556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:45.954846 22556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:45.955054 22556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:45.963667 22556 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.252:46229
I20260620 14:50:45.963709 22623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.252:46229 every 8 connection(s)
I20260620 14:50:45.964879 22556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-2/data/info.pb
I20260620 14:50:45.968237 22624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:45.972494 22624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:45.974113 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 22556
I20260620 14:50:45.974336 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966871461349-9079-0/minicluster-data/master-2/wal/instance
I20260620 14:50:45.974743 9079 external_mini_cluster.cc:1428] Setting key 392977b2bc90edfc011d5b4704a686b1
Jun 20 14:50:45 dist-test-slave-65tz krb5kdc[22392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967045, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.252@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
Jun 20 14:50:45 dist-test-slave-65tz krb5kdc[22392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.252@KRBTEST.COM
I20260620 14:50:45.988062 22476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.975048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60922 (local address 127.8.221.254:38395)
0620 14:50:45.975194 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:45.975199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:45.976519 (+ 1320us) server_negotiation.cc:407] Connection header received
0620 14:50:45.977575 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:45.977578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:45.977638 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:45.977749 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:45.978929 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.979457 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.980518 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.980739 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.980854 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:45.983984 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:45.983987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:45.983989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:45.984015 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:45.985681 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:45.986373 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:45.986378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:45.986380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:45.986465 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:45.986829 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:45.986832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:45.986834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:45.987048 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:45.987153 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:45.987759 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 14:50:45.987880 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:50:45.989365 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.975680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38395 (local address 127.0.0.1:60922)
0620 14:50:45.976359 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.976414 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:45.977347 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:45.977833 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:45.977843 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:45.978246 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:45.978780 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:45.978795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.979570 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.979574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:45.980385 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:45.980394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.980904 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.980908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:45.981169 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:45.981929 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:45.981956 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:45.983784 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:45.985845 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:45.985858 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:45.985871 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:45.986262 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:45.986560 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:45.986565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:45.986567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:45.986725 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:45.987166 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:45.987175 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:45.987524 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 14:50:45.987708 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 14:50:45.988058 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":88,"thread_start_us":212,"threads_started":1}
I20260620 14:50:45.989373 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.975680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.8.221.252:46229)
0620 14:50:45.976914 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:45.976924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:45.976983 (+ 59us) server_negotiation.cc:407] Connection header received
0620 14:50:45.977370 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:45.977398 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:45.977632 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:45.977927 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:45.978901 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.979895 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.980796 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.981321 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.981452 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:45.984300 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:45.984307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:45.984321 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:45.984344 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:45.986238 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:45.986689 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:45.986701 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:45.986702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:45.986794 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:45.987089 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:45.987092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:45.987094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:45.987513 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:45.987746 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:45.988045 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 14:50:45.988219 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":1096,"spinlock_wait_cycles":71680,"thread_start_us":212,"threads_started":1}
I20260620 14:50:45.990864 22624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:50:45 dist-test-slave-65tz krb5kdc[22392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967045, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.252@KRBTEST.COM for kudu/127.8.221.253@KRBTEST.COM
Jun 20 14:50:45 dist-test-slave-65tz krb5kdc[22392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:50:46.000780 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.991430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:45249 (local address 127.0.0.1:33240)
0620 14:50:45.991553 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.991569 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:45.991669 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:45.991980 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:45.991983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:45.992158 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:45.992303 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:45.992310 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.993256 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.993262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:45.993939 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:45.993948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.994398 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.994402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:45.994504 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:45.995027 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:45.995048 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:45.996718 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:45.998855 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:45.998860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:45.998862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:45.999128 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:45.999483 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:45.999487 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:45.999489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:45.999594 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:46.000137 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:46.000140 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:46.000239 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 14:50:46.000340 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:50:46.000527 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 14:50:46.000801 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.991596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33240 (local address 127.8.221.253:45249)
0620 14:50:45.991784 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:45.991788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:45.991800 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:50:45.991836 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:45.991838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:45.991879 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:45.991966 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:45.992433 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.993152 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.994061 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.994280 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.994405 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:45.996896 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:45.996901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:45.996904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:45.996939 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:45.998707 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:45.999286 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:45.999289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:45.999291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:45.999375 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:45.999744 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:45.999747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:45.999748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:45.999972 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:46.000139 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:46.000465 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 14:50:46.000584 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:50:46.001916 22476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.992407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60924 (local address 127.8.221.254:38395)
0620 14:50:45.992543 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:45.992548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:45.992636 (+ 88us) server_negotiation.cc:407] Connection header received
0620 14:50:45.992723 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:45.992726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:45.992765 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:45.992852 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:45.993508 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.994108 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.994925 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:45.995149 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:45.995272 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:45.997969 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:45.997975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:45.997979 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:45.998016 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:45.999914 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:46.000459 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:46.000463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:46.000465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:46.000557 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:46.000926 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:46.000931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:46.000935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:46.001170 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:46.001463 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:46.001540 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 14:50:46.001685 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:50:46.001942 22624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:50:46 dist-test-slave-65tz krb5kdc[22392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967045, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.252@KRBTEST.COM for kudu/127.8.221.252@KRBTEST.COM
Jun 20 14:50:46 dist-test-slave-65tz krb5kdc[22392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.253@KRBTEST.COM
I20260620 14:50:46.013396 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:46.002681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:46229 (local address 127.0.0.1:33610)
0620 14:50:46.002877 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:46.002892 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:46.002962 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:46.003749 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:46.003752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:46.003902 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:46.003996 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:46.004001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.004806 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.004812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:46.005328 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:46.005335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.005899 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.005903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:46.005983 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:46.006593 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:46.006609 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:46.008068 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:46.011289 (+ 3221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:46.011293 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:46.011295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:46.011685 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:46.012099 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:46.012102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:46.012103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:46.012240 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:46.012693 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:46.012699 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:46.012815 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 14:50:46.012957 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 14:50:46.013193 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 14:50:46.013567 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:46.003570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33244 (local address 127.8.221.253:45249)
0620 14:50:46.003722 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:46.003728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:46.003764 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:50:46.003930 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:46.003933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:46.003974 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:46.004065 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:46.004709 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.005219 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.006058 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.006417 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.006534 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:46.009258 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:46.009265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:46.009269 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:46.009312 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:46.011245 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:46.011842 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:46.011846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:46.011848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:46.011982 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:46.012399 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:46.012403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:46.012404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:46.012651 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:46.012967 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:46.013267 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 14:50:46.013411 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:50:46.013971 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:46.002923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.8.221.252:46229)
0620 14:50:46.003050 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:46.003054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:46.003069 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:50:46.003112 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:46.003115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:46.003172 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:46.003267 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:46.004128 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.004674 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.005468 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.005738 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.005897 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:46.008227 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:46.008237 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:46.008240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:46.008275 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:46.010900 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:46.011842 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:46.011846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:46.011848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:46.011970 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:46.012354 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:46.012360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:46.012364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:46.012583 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:46.012666 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:46.013641 (+ 975us) server_negotiation.cc:299] Negotiation successful
0620 14:50:46.013764 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:50:46.016753 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295: Bootstrap starting.
I20260620 14:50:46.019567 22624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:46.020385 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:46.020411 22624 log.cc:824] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:46.022521 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295: No bootstrap required, opened a new log
I20260620 14:50:46.025677 22624 raft_consensus.cc:348] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } } peers { permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } } peers { permanent_uuid: "aca4f423e004496ebc7a8760d5d66295" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } }
I20260620 14:50:46.026007 22624 raft_consensus.cc:374] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:46.026129 22624 raft_consensus.cc:729] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca4f423e004496ebc7a8760d5d66295, State: Initialized, Role: FOLLOWER
I20260620 14:50:46.026666 22624 consensus_queue.cc:260] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } } peers { permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } } peers { permanent_uuid: "aca4f423e004496ebc7a8760d5d66295" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } }
I20260620 14:50:46.028116 22624 sys_catalog.cc:564] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:46.033048 22637 sys_catalog.cc:455] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } } peers { permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } } peers { permanent_uuid: "aca4f423e004496ebc7a8760d5d66295" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } } }
I20260620 14:50:46.034194 22476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:46.022761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.8.221.254:38395)
0620 14:50:46.022918 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:46.022925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:46.023659 (+ 734us) server_negotiation.cc:407] Connection header received
0620 14:50:46.023832 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:46.023835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:46.023895 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:46.023984 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:46.024736 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.025304 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.026326 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.026639 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.026750 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:46.029040 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:46.029043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:46.029047 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:46.029072 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:46.031481 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:46.031975 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:46.031979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:46.031981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:46.032063 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:46.032386 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:46.032390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:46.032391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:46.032904 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:46.033422 (+ 518us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:46.033830 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 14:50:46.033981 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 14:50:46.034441 22648 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:50:46.034523 22648 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:50:46.033233 22637 sys_catalog.cc:458] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:46.036023 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:46.023299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33612 (local address 127.8.221.252:46229)
0620 14:50:46.023424 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:46.023427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:46.026599 (+ 3172us) server_negotiation.cc:407] Connection header received
0620 14:50:46.026766 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:46.026769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:46.026823 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:46.026874 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:46.027916 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.028635 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.029929 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.030255 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.030338 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:46.032225 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:46.032229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:46.032231 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:46.032256 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:46.033993 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:46.034484 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:46.034487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:46.034489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:46.034585 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:46.034900 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:46.034904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:46.034906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:46.035200 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:46.035330 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:46.035716 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 14:50:46.035862 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:50:46.032441 22556 master_runner.cc:186] Error getting master registration for 127.8.221.254:38395: OK, instance_id { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" instance_seqno: 1781967045441531 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:50:46.037935 22556 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:50:46.040915 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:46.023206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33246 (local address 127.8.221.253:45249)
0620 14:50:46.023332 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:46.023337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:46.026160 (+ 2823us) server_negotiation.cc:407] Connection header received
0620 14:50:46.026435 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:46.026438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:46.026499 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:46.026580 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:46.027522 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.028316 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.029463 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.029879 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.029962 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:46.037561 (+ 7599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:46.037565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:46.037568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:46.037601 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:46.039140 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:46.039667 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:46.039670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:46.039671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:46.039742 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:46.040047 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:46.040049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:46.040050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:46.040243 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:46.040367 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:46.040658 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 14:50:46.040755 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:50:46.080948 22549 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } attempt: 4
Jun 20 14:50:46 dist-test-slave-65tz krb5kdc[22392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967045, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.253@KRBTEST.COM for kudu/127.8.221.252@KRBTEST.COM
I20260620 14:50:46.086771 22473 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } attempt: 4
I20260620 14:50:46.090940 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:46.081443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:46229 (local address 127.0.0.1:33620)
0620 14:50:46.081629 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:46.081645 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:46.081726 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:46.082051 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:46.082055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:46.082275 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:46.082396 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:46.082402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.083178 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.083181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:46.084054 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:46.084064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.084568 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.084572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:46.084659 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:46.085225 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:46.085239 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:46.086867 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:46.089037 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:46.089040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:46.089043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:46.089347 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:46.089696 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:46.089699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:46.089701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:46.089830 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:46.090303 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:46.090307 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:46.090428 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 14:50:46.090571 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 14:50:46.090746 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 14:50:46.090945 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:46.081565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33620 (local address 127.8.221.252:46229)
0620 14:50:46.081723 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:46.081728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:46.081745 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:50:46.081851 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:46.081854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:46.081912 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:46.081996 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:46.082516 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.083063 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.084185 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.084455 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.084541 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:46.087065 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:46.087069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:46.087073 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:46.087104 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:46.088898 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:46.089473 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:46.089477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:46.089479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:46.089570 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:46.089933 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:46.089936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:46.089938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:46.090179 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:46.090283 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:46.090649 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 14:50:46.090798 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":63}
Jun 20 14:50:46 dist-test-slave-65tz krb5kdc[22392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967045, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for kudu/127.8.221.252@KRBTEST.COM
I20260620 14:50:46.095183 22549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302: Bootstrap starting.
I20260620 14:50:46.097040 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:46.087276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:46229 (local address 127.0.0.1:33630)
0620 14:50:46.087502 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:46.087519 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:46.087607 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:46.087980 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:46.088229 (+ 249us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:46.088420 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:46.088545 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:46.088551 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.089559 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.089563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:46.090338 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:46.090353 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.091086 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.091092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:46.091227 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:46.091947 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:46.091962 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:46.093456 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:46.095496 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:46.095500 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:46.095502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:46.095735 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:46.096027 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:46.096030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:46.096032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:46.096124 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:46.096535 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:46.096538 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:46.096637 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 14:50:46.096718 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:50:46.096856 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260620 14:50:46.097728 22650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:46.087324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33630 (local address 127.8.221.252:46229)
0620 14:50:46.087764 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:46.087772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:46.087785 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:50:46.087813 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:46.087815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:46.087850 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:46.087963 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:46.088681 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.089397 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.090477 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:46.090915 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:46.091460 (+ 545us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:46.093615 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:46.093619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:46.093622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:46.093646 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:46.095371 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:46.095851 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:46.095854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:46.095856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:46.095929 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:46.096231 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:46.096233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:46.096235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:46.096430 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:46.097360 (+ 930us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:46.097434 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 14:50:46.097552 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":365,"thread_start_us":151,"threads_started":1}
I20260620 14:50:46.098668 22549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:46.099673 22549 log.cc:824] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:46.100960 22473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2: Bootstrap starting.
I20260620 14:50:46.103513 22473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:46.104434 22473 log.cc:824] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:46.104790 22549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302: No bootstrap required, opened a new log
I20260620 14:50:46.106757 22473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2: No bootstrap required, opened a new log
I20260620 14:50:46.109270 22549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } } peers { permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } } peers { permanent_uuid: "aca4f423e004496ebc7a8760d5d66295" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } }
I20260620 14:50:46.109527 22549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:46.109586 22549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aded7d26e9f425685994de6ecba0302, State: Initialized, Role: FOLLOWER
I20260620 14:50:46.110188 22549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } } peers { permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } } peers { permanent_uuid: "aca4f423e004496ebc7a8760d5d66295" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } }
I20260620 14:50:46.110846 22653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } } peers { permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } } peers { permanent_uuid: "aca4f423e004496ebc7a8760d5d66295" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } } }
I20260620 14:50:46.110881 22473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } } peers { permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } } peers { permanent_uuid: "aca4f423e004496ebc7a8760d5d66295" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } }
I20260620 14:50:46.111089 22473 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:46.111157 22473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b44b03c3844be0bc949cf620199fc2, State: Initialized, Role: FOLLOWER
I20260620 14:50:46.111246 22653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:46.111687 22473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } } peers { permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } } peers { permanent_uuid: "aca4f423e004496ebc7a8760d5d66295" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } }
I20260620 14:50:46.112129 22549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:46.112370 22654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } } peers { permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } } peers { permanent_uuid: "aca4f423e004496ebc7a8760d5d66295" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } } }
I20260620 14:50:46.112650 22654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:46.113817 22473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:50:46.119668 22675 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:50:46.119900 22675 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 14:50:46.121366 22676 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:50:46.121464 22676 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:50:46.126742 22405 master_runner.cc:263] No leader master found from master 83b44b03c3844be0bc949cf620199fc2
I20260620 14:50:46.126860 22405 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:50:46.127810 22479 master_runner.cc:263] No leader master found from master 3aded7d26e9f425685994de6ecba0302
I20260620 14:50:46.127895 22479 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 14:50:46.175438 22502 tablet.cc:2369] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:50:46.187985 22426 tablet.cc:2369] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:50:46.206810 22577 tablet.cc:2369] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:50:46.213184 22637 raft_consensus.cc:482] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:50:46.213496 22637 raft_consensus.cc:504] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } } peers { permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } } peers { permanent_uuid: "aca4f423e004496ebc7a8760d5d66295" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } }
I20260620 14:50:46.215284 22637 leader_election.cc:288] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b44b03c3844be0bc949cf620199fc2 (127.8.221.254:38395), 3aded7d26e9f425685994de6ecba0302 (127.8.221.253:45249)
I20260620 14:50:46.216362 22447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aca4f423e004496ebc7a8760d5d66295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b44b03c3844be0bc949cf620199fc2" is_pre_election: true
I20260620 14:50:46.216789 22447 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aca4f423e004496ebc7a8760d5d66295 in term 0.
I20260620 14:50:46.217480 22573 leader_election.cc:302] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b44b03c3844be0bc949cf620199fc2, aca4f423e004496ebc7a8760d5d66295; no voters:
I20260620 14:50:46.217968 22637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:50:46.218060 22637 raft_consensus.cc:482] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:50:46.218104 22637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:46.219732 22637 raft_consensus.cc:504] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } } peers { permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } } peers { permanent_uuid: "aca4f423e004496ebc7a8760d5d66295" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } }
I20260620 14:50:46.220700 22447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aca4f423e004496ebc7a8760d5d66295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b44b03c3844be0bc949cf620199fc2"
I20260620 14:50:46.220891 22447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:46.222546 22523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aca4f423e004496ebc7a8760d5d66295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aded7d26e9f425685994de6ecba0302" is_pre_election: true
I20260620 14:50:46.223013 22523 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aca4f423e004496ebc7a8760d5d66295 in term 0.
I20260620 14:50:46.224490 22447 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aca4f423e004496ebc7a8760d5d66295 in term 1.
I20260620 14:50:46.224893 22573 leader_election.cc:302] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b44b03c3844be0bc949cf620199fc2, aca4f423e004496ebc7a8760d5d66295; no voters:
I20260620 14:50:46.227936 22637 leader_election.cc:288] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [CANDIDATE]: Term 1 election: Requested vote from peers 83b44b03c3844be0bc949cf620199fc2 (127.8.221.254:38395), 3aded7d26e9f425685994de6ecba0302 (127.8.221.253:45249)
I20260620 14:50:46.228117 22637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:46.228200 22637 raft_consensus.cc:686] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [term 1 LEADER]: Becoming Leader. State: Replica: aca4f423e004496ebc7a8760d5d66295, State: Running, Role: LEADER
I20260620 14:50:46.228294 22523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aca4f423e004496ebc7a8760d5d66295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aded7d26e9f425685994de6ecba0302"
I20260620 14:50:46.228430 22523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:46.228619 22637 consensus_queue.cc:237] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } } peers { permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } } peers { permanent_uuid: "aca4f423e004496ebc7a8760d5d66295" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } }
I20260620 14:50:46.230226 22523 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aca4f423e004496ebc7a8760d5d66295 in term 1.
I20260620 14:50:46.230562 22637 sys_catalog.cc:455] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aca4f423e004496ebc7a8760d5d66295. Latest consensus state: current_term: 1 leader_uuid: "aca4f423e004496ebc7a8760d5d66295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } } peers { permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } } peers { permanent_uuid: "aca4f423e004496ebc7a8760d5d66295" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } } }
I20260620 14:50:46.230693 22637 sys_catalog.cc:458] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:46.231158 22681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:46.232306 22681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:46.238826 22523 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302 [term 1 FOLLOWER]: Refusing update from remote peer aca4f423e004496ebc7a8760d5d66295: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:46.239053 22447 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2 [term 1 FOLLOWER]: Refusing update from remote peer aca4f423e004496ebc7a8760d5d66295: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:46.239727 22679 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:46.239938 22637 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:46.244729 22654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aca4f423e004496ebc7a8760d5d66295. Latest consensus state: current_term: 1 leader_uuid: "aca4f423e004496ebc7a8760d5d66295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } } peers { permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } } peers { permanent_uuid: "aca4f423e004496ebc7a8760d5d66295" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } } }
I20260620 14:50:46.244884 22654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:46.245179 22653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aca4f423e004496ebc7a8760d5d66295. Latest consensus state: current_term: 1 leader_uuid: "aca4f423e004496ebc7a8760d5d66295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } } peers { permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } } peers { permanent_uuid: "aca4f423e004496ebc7a8760d5d66295" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } } }
I20260620 14:50:46.245333 22653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:46.248786 22637 sys_catalog.cc:455] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aca4f423e004496ebc7a8760d5d66295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } } peers { permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } } peers { permanent_uuid: "aca4f423e004496ebc7a8760d5d66295" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } } }
I20260620 14:50:46.248948 22637 sys_catalog.cc:458] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:46.251184 22679 sys_catalog.cc:455] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aca4f423e004496ebc7a8760d5d66295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } } peers { permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } } peers { permanent_uuid: "aca4f423e004496ebc7a8760d5d66295" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } } }
I20260620 14:50:46.251358 22679 sys_catalog.cc:458] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:46.252619 22682 mvcc.cc:205] Tried to move back new op lower bound from 7298937021385052160 to 7298937021355724800. Current Snapshot: MvccSnapshot[applied={T|T < 7298937021385052160}]
I20260620 14:50:46.252755 22683 mvcc.cc:205] Tried to move back new op lower bound from 7298937021385052160 to 7298937021355724800. Current Snapshot: MvccSnapshot[applied={T|T < 7298937021385052160}]
I20260620 14:50:46.253314 22684 mvcc.cc:205] Tried to move back new op lower bound from 7298937021385052160 to 7298937021355724800. Current Snapshot: MvccSnapshot[applied={T|T < 7298937021385052160}]
I20260620 14:50:46.253726 22653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aca4f423e004496ebc7a8760d5d66295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } } peers { permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } } peers { permanent_uuid: "aca4f423e004496ebc7a8760d5d66295" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } } }
I20260620 14:50:46.253929 22653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:46.253952 22654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aca4f423e004496ebc7a8760d5d66295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83b44b03c3844be0bc949cf620199fc2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38395 } } peers { permanent_uuid: "3aded7d26e9f425685994de6ecba0302" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 45249 } } peers { permanent_uuid: "aca4f423e004496ebc7a8760d5d66295" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 46229 } } }
I20260620 14:50:46.254101 22654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:46.256204 22681 catalog_manager.cc:1347] Generated new cluster ID: 45c7d723170446348b8a500a7d1b5e27
I20260620 14:50:46.256280 22681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:46.274379 22681 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:46.275904 22681 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:46.287087 22681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aca4f423e004496ebc7a8760d5d66295: Generated new TSK 0
I20260620 14:50:46.288321 22681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:47.121908 22675 catalog_manager.cc:1259] Loaded cluster ID: 45c7d723170446348b8a500a7d1b5e27
I20260620 14:50:47.122040 22675 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2: loading cluster ID for follower catalog manager: success
I20260620 14:50:47.122991 22676 catalog_manager.cc:1259] Loaded cluster ID: 45c7d723170446348b8a500a7d1b5e27
I20260620 14:50:47.123147 22676 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302: loading cluster ID for follower catalog manager: success
I20260620 14:50:47.124629 22675 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2: acquiring CA information for follower catalog manager: success
I20260620 14:50:47.125226 22676 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302: acquiring CA information for follower catalog manager: success
I20260620 14:50:47.126448 22675 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 83b44b03c3844be0bc949cf620199fc2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:50:47.126479 22676 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3aded7d26e9f425685994de6ecba0302: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:50:47.357877 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 22405
I20260620 14:50:47.364998 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 22479
I20260620 14:50:47.371580 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 22556
2026-06-20T14:50:47Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3303 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:47 dist-test-slave-65tz krb5kdc[22696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:47 dist-test-slave-65tz krb5kdc[22696](info): set up 2 sockets
Jun 20 14:50:47 dist-test-slave-65tz krb5kdc[22696](info): commencing operation
krb5kdc: starting...
W20260620 14:50:48.438627 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:50:48 dist-test-slave-65tz krb5kdc[22696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:50:48.574968 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38139
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:46069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:38139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:48.703752 22709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:48.704092 22709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:48.704138 22709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:48.708664 22709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:48.708767 22709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:48.708789 22709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:48.708808 22709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:48.708824 22709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:48.714907 22709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:46069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:38139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:48.716390 22709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:48.717720 22709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:48.728156 22714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:48.728147 22717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:48.728148 22715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:48.728562 22709 server_base.cc:1034] running on GCE node
I20260620 14:50:48.729519 22709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:48.730983 22709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:48.732200 22709 hybrid_clock.cc:648] HybridClock initialized: now 1781967048732158 us; error 65 us; skew 500 ppm
Jun 20 14:50:48 dist-test-slave-65tz krb5kdc[22696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967048, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:48.735422 22709 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:50:48.736574 22709 webserver.cc:460] Webserver started at http://127.8.221.254:45515/ using document root <none> and password file <none>
I20260620 14:50:48.737258 22709 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:48.737355 22709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:48.737655 22709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:48.740554 22709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "9ca1ae4d98604149a24aa0cee49e28c0"
format_stamp: "Formatted at 2026-06-20 14:50:48 on dist-test-slave-65tz"
server_key: "c6d53861ef34613574560a254e43bdfa"
server_key_iv: "1c14e4e6a41041b8e85492cfe4682c9a"
server_key_version: "encryptionkey@0"
I20260620 14:50:48.741247 22709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "9ca1ae4d98604149a24aa0cee49e28c0"
format_stamp: "Formatted at 2026-06-20 14:50:48 on dist-test-slave-65tz"
server_key: "c6d53861ef34613574560a254e43bdfa"
server_key_iv: "1c14e4e6a41041b8e85492cfe4682c9a"
server_key_version: "encryptionkey@0"
I20260620 14:50:48.745961 22709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 14:50:48.748890 22724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:48.750633 22709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:48.750813 22709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "9ca1ae4d98604149a24aa0cee49e28c0"
format_stamp: "Formatted at 2026-06-20 14:50:48 on dist-test-slave-65tz"
server_key: "c6d53861ef34613574560a254e43bdfa"
server_key_iv: "1c14e4e6a41041b8e85492cfe4682c9a"
server_key_version: "encryptionkey@0"
I20260620 14:50:48.750955 22709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:48.774781 22709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:48.779423 22709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:48.779685 22709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:48.789738 22709 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:38139
I20260620 14:50:48.789883 22776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:38139 every 8 connection(s)
I20260620 14:50:48.790910 22709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:48.793079 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 22709
I20260620 14:50:48.793242 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:50:48.793524 9079 external_mini_cluster.cc:1428] Setting key ecff124bc51e4b1f5e7c200f646997d0
I20260620 14:50:48.795118 22777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:48.804054 22777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0: Bootstrap starting.
Jun 20 14:50:48 dist-test-slave-65tz krb5kdc[22696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:50:48.807710 22777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:48.808791 22777 log.cc:824] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:48.811807 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:48.795150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60052 (local address 127.8.221.254:38139)
0620 14:50:48.795659 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:48.795676 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:48.795716 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:50:48.796444 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:48.796462 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:48.796837 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:48.797144 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:48.798480 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:48.799593 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:48.800977 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:48.801533 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:48.801661 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:48.805272 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:48.805281 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:48.805334 (+ 53us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:48.805385 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:48.807744 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:48.808375 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:48.808382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:48.808390 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:48.808569 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:48.808957 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:48.808961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:48.808963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:48.809591 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:48.809778 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:48.810266 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 14:50:48.810673 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":139,"threads_started":1}
I20260620 14:50:48.811906 22777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0: No bootstrap required, opened a new log
I20260620 14:50:48.815469 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:48.817049 22777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca1ae4d98604149a24aa0cee49e28c0" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38139 } }
I20260620 14:50:48.817452 22777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:48.817555 22777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca1ae4d98604149a24aa0cee49e28c0, State: Initialized, Role: FOLLOWER
I20260620 14:50:48.818184 22777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca1ae4d98604149a24aa0cee49e28c0" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38139 } }
I20260620 14:50:48.818377 22777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:48.818498 22777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:48.818635 22777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:48.819877 22777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca1ae4d98604149a24aa0cee49e28c0" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38139 } }
I20260620 14:50:48.820328 22777 leader_election.cc:302] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ca1ae4d98604149a24aa0cee49e28c0; no voters:
I20260620 14:50:48.820752 22777 leader_election.cc:288] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:48.820967 22782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:48.821316 22782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0 [term 1 LEADER]: Becoming Leader. State: Replica: 9ca1ae4d98604149a24aa0cee49e28c0, State: Running, Role: LEADER
I20260620 14:50:48.821710 22782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca1ae4d98604149a24aa0cee49e28c0" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38139 } }
I20260620 14:50:48.822526 22777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:48.823647 22784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ca1ae4d98604149a24aa0cee49e28c0. Latest consensus state: current_term: 1 leader_uuid: "9ca1ae4d98604149a24aa0cee49e28c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca1ae4d98604149a24aa0cee49e28c0" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38139 } } }
I20260620 14:50:48.823604 22783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ca1ae4d98604149a24aa0cee49e28c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca1ae4d98604149a24aa0cee49e28c0" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38139 } } }
I20260620 14:50:48.823777 22784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:48.823777 22783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:48.824229 22791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:48.827392 22791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:48.834357 22791 catalog_manager.cc:1347] Generated new cluster ID: 622d8e6ad583434eb4aae3b7cec9f241
I20260620 14:50:48.834438 22791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:48.856472 22791 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:48.858186 22791 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:48.864431 22791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ca1ae4d98604149a24aa0cee49e28c0: Generated new TSK 0
I20260620 14:50:48.865298 22791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:48.871769 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 22709
2026-06-20T14:50:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1498 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:48 dist-test-slave-65tz krb5kdc[22804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:48 dist-test-slave-65tz krb5kdc[22804](info): set up 2 sockets
Jun 20 14:50:48 dist-test-slave-65tz krb5kdc[22804](info): commencing operation
krb5kdc: starting...
W20260620 14:50:49.926914 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.006s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:50:50 dist-test-slave-65tz krb5kdc[22804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:50:50.062884 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:43407
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:43407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:50.201190 22817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:50.201572 22817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:50.201654 22817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:50.206113 22817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:50.206229 22817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:50.206266 22817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:50.206305 22817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:50.206363 22817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:50.212886 22817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:43407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:43407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:50.214454 22817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:50.215790 22817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:50:50.227433 22817 server_base.cc:1034] running on GCE node
W20260620 14:50:50.227380 22822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:50.227380 22825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:50.227385 22823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:50.228410 22817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:50.229758 22817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:50.231014 22817 hybrid_clock.cc:648] HybridClock initialized: now 1781967050230995 us; error 38 us; skew 500 ppm
Jun 20 14:50:50 dist-test-slave-65tz krb5kdc[22804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967050, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:50.238518 22817 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:50:50.239636 22817 webserver.cc:460] Webserver started at http://127.8.221.254:41951/ using document root <none> and password file <none>
I20260620 14:50:50.240283 22817 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:50.240355 22817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:50.240571 22817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:50.242959 22817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "1621fd4489704ef9a2f1f48d87190fbb"
format_stamp: "Formatted at 2026-06-20 14:50:50 on dist-test-slave-65tz"
server_key: "0deb52d1f9e8b26ae72a632ba2444dad"
server_key_iv: "b08d1301e48649c1c4f5bd3d575207f1"
server_key_version: "encryptionkey@0"
I20260620 14:50:50.243569 22817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "1621fd4489704ef9a2f1f48d87190fbb"
format_stamp: "Formatted at 2026-06-20 14:50:50 on dist-test-slave-65tz"
server_key: "0deb52d1f9e8b26ae72a632ba2444dad"
server_key_iv: "b08d1301e48649c1c4f5bd3d575207f1"
server_key_version: "encryptionkey@0"
I20260620 14:50:50.248108 22817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:50:50.251083 22832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:50.252637 22817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:50.252816 22817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "1621fd4489704ef9a2f1f48d87190fbb"
format_stamp: "Formatted at 2026-06-20 14:50:50 on dist-test-slave-65tz"
server_key: "0deb52d1f9e8b26ae72a632ba2444dad"
server_key_iv: "b08d1301e48649c1c4f5bd3d575207f1"
server_key_version: "encryptionkey@0"
I20260620 14:50:50.252942 22817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:50.276053 22817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:50.279879 22817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:50.280148 22817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:50.289225 22817 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:43407
I20260620 14:50:50.289618 22884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:43407 every 8 connection(s)
I20260620 14:50:50.290426 22817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:50.291802 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 22817
I20260620 14:50:50.291970 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:50:50.292253 9079 external_mini_cluster.cc:1428] Setting key 27c178fbd3c29840cd004901886e6787
I20260620 14:50:50.295316 22885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 14:50:50 dist-test-slave-65tz krb5kdc[22804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:50:50.303442 22885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb: Bootstrap starting.
I20260620 14:50:50.306749 22885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:50.307825 22885 log.cc:824] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:50.308871 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:50.294158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.8.221.254:43407)
0620 14:50:50.294800 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:50.294819 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:50.294864 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:50:50.295735 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:50.295757 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:50.296163 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:50.296544 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:50.297361 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:50.298473 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:50.299393 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:50.299908 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:50.300134 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:50.303171 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:50.303177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:50.303193 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:50.303225 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:50.305129 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:50.305908 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:50.305913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:50.305918 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:50.306044 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:50.306396 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:50.306398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:50.306399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:50.306845 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:50.307073 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:50.307504 (+ 431us) server_negotiation.cc:299] Negotiation successful
0620 14:50:50.307851 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":209,"threads_started":1}
I20260620 14:50:50.311014 22885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb: No bootstrap required, opened a new log
I20260620 14:50:50.315404 22885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1621fd4489704ef9a2f1f48d87190fbb" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43407 } }
I20260620 14:50:50.315762 22885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:50.315843 22885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1621fd4489704ef9a2f1f48d87190fbb, State: Initialized, Role: FOLLOWER
I20260620 14:50:50.316417 22885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1621fd4489704ef9a2f1f48d87190fbb" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43407 } }
I20260620 14:50:50.316584 22885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:50.316671 22885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:50.316780 22885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:50.318089 22885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1621fd4489704ef9a2f1f48d87190fbb" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43407 } }
I20260620 14:50:50.318490 22885 leader_election.cc:302] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1621fd4489704ef9a2f1f48d87190fbb; no voters:
I20260620 14:50:50.318886 22885 leader_election.cc:288] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:50.319074 22890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:50.319415 22890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 1 LEADER]: Becoming Leader. State: Replica: 1621fd4489704ef9a2f1f48d87190fbb, State: Running, Role: LEADER
I20260620 14:50:50.319872 22890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1621fd4489704ef9a2f1f48d87190fbb" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43407 } }
I20260620 14:50:50.320457 22885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:50.322284 22892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1621fd4489704ef9a2f1f48d87190fbb. Latest consensus state: current_term: 1 leader_uuid: "1621fd4489704ef9a2f1f48d87190fbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1621fd4489704ef9a2f1f48d87190fbb" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43407 } } }
I20260620 14:50:50.322261 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1621fd4489704ef9a2f1f48d87190fbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1621fd4489704ef9a2f1f48d87190fbb" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43407 } } }
I20260620 14:50:50.322531 22892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:50.322551 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [sys.catalog]: This master's current role is: LEADER
W20260620 14:50:50.326241 22904 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:50:50.326392 22904 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:50:50.326558 22905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:50.327632 22905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:50.334513 22905 catalog_manager.cc:1347] Generated new cluster ID: 43eae1cb22eb4f96a1618abe483e3adf
I20260620 14:50:50.334622 22905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:50.367053 22905 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:50.368207 22905 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:50.375339 22905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb: Generated new TSK 0
I20260620 14:50:50.376116 22905 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
I20260620 14:50:50.412693 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:43407
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:50.531859 22911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:50.532204 22911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:50.532255 22911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:50.536772 22911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:50.536854 22911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:50.536960 22911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:50:50.543030 22911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:43407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:50.544409 22911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:50.545665 22911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:50.557051 22916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:50.557195 22917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:50.557054 22919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:50.557531 22911 server_base.cc:1034] running on GCE node
I20260620 14:50:50.558126 22911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:50.558992 22911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:50.560201 22911 hybrid_clock.cc:648] HybridClock initialized: now 1781967050560191 us; error 39 us; skew 500 ppm
Jun 20 14:50:50 dist-test-slave-65tz krb5kdc[22804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967050, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:50.563436 22911 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:50:50.564486 22911 webserver.cc:460] Webserver started at http://127.8.221.193:34955/ using document root <none> and password file <none>
I20260620 14:50:50.565086 22911 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:50.565147 22911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:50.565398 22911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:50.567776 22911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "aff3fa9f4ed64017aad4d7f2804c0869"
format_stamp: "Formatted at 2026-06-20 14:50:50 on dist-test-slave-65tz"
server_key: "f1083f14ef0729e5b606d9c8d91f9c95"
server_key_iv: "7db513b06b2198e38e54e8c4f08c3007"
server_key_version: "encryptionkey@0"
I20260620 14:50:50.568418 22911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "aff3fa9f4ed64017aad4d7f2804c0869"
format_stamp: "Formatted at 2026-06-20 14:50:50 on dist-test-slave-65tz"
server_key: "f1083f14ef0729e5b606d9c8d91f9c95"
server_key_iv: "7db513b06b2198e38e54e8c4f08c3007"
server_key_version: "encryptionkey@0"
I20260620 14:50:50.572908 22911 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:50:50.575786 22926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:50.577303 22911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:50:50.577483 22911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "aff3fa9f4ed64017aad4d7f2804c0869"
format_stamp: "Formatted at 2026-06-20 14:50:50 on dist-test-slave-65tz"
server_key: "f1083f14ef0729e5b606d9c8d91f9c95"
server_key_iv: "7db513b06b2198e38e54e8c4f08c3007"
server_key_version: "encryptionkey@0"
I20260620 14:50:50.577620 22911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:50.605973 22911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:50.609983 22911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:50.610242 22911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:50.611014 22911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:50.612016 22911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:50.612092 22911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:50.612195 22911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:50.612239 22911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:50.624187 22911 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:40425
I20260620 14:50:50.624208 23039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:40425 every 8 connection(s)
I20260620 14:50:50.625308 22911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:50:50.630021 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 22911
I20260620 14:50:50.630177 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:50:50.630554 9079 external_mini_cluster.cc:1428] Setting key db22153ec52d03cf9c2cf3e2f335b6bf
Jun 20 14:50:50 dist-test-slave-65tz krb5kdc[22804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967050, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.194@KRBTEST.COM; defaulting to no policy
I20260620 14:50:50.642988 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:50.627586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:40371 (local address 127.8.221.254:43407)
0620 14:50:50.627748 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:50.627752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:50.628652 (+ 900us) server_negotiation.cc:407] Connection header received
0620 14:50:50.630577 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:50.630581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:50.630689 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:50.630790 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:50.632310 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:50.632860 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:50.634787 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:50.635233 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:50.635377 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:50.638443 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:50.638446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:50.638448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:50.638473 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:50.640279 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:50.640998 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:50.641000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:50.641002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:50.641119 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:50.641586 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:50.641589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:50.641590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:50.641772 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:50.641972 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:50.642740 (+ 768us) server_negotiation.cc:299] Negotiation successful
0620 14:50:50.642855 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:50:50.644024 23042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:50.627882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:40371)
0620 14:50:50.628462 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:50.628516 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:50.630246 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:50.631067 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:50.631075 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:50.631506 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:50.632118 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:50.632132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:50.633014 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:50.633021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:50.634632 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:50.634642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:50.635431 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:50.635434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:50.635594 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:50.636350 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:50.636368 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:50.638227 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:50.640455 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:50.640463 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:50.640485 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:50.640874 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:50.641228 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:50.641233 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:50.641239 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:50.641455 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:50.641995 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:50.642001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:50.642409 (+ 408us) client_negotiation.cc:769] Sending connection context
0620 14:50:50.642655 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 14:50:50.642988 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":161,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.194@KRBTEST.COM" created.
I20260620 14:50:50.645733 23040 heartbeater.cc:347] Connected to a master server at 127.8.221.254:43407
I20260620 14:50:50.646081 23040 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:50.646672 23040 heartbeater.cc:511] Master 127.8.221.254:43407 requested a full tablet report, sending...
I20260620 14:50:50.648293 22849 ts_manager.cc:194] Registered new tserver with Master: aff3fa9f4ed64017aad4d7f2804c0869 (127.8.221.193:40425)
I20260620 14:50:50.649415 22849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:40371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
I20260620 14:50:50.665357 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:43407
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:50.778985 23045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:50.779276 23045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:50.779323 23045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:50.784153 23045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:50.784215 23045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:50.784319 23045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:50:50.791497 23045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:43407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:50.792639 23045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:50.793640 23045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:50.804226 23051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:50.804261 23050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:50.804471 23045 server_base.cc:1034] running on GCE node
W20260620 14:50:50.804261 23053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:50.805197 23045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:50.806116 23045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:50.807339 23045 hybrid_clock.cc:648] HybridClock initialized: now 1781967050807293 us; error 53 us; skew 500 ppm
Jun 20 14:50:50 dist-test-slave-65tz krb5kdc[22804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967050, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:50.810901 23045 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:50:50.812109 23045 webserver.cc:460] Webserver started at http://127.8.221.194:37053/ using document root <none> and password file <none>
I20260620 14:50:50.812839 23045 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:50.812906 23045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:50.813156 23045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:50.815637 23045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "a9301e8263f34d22a8ad809ee4a2ed48"
format_stamp: "Formatted at 2026-06-20 14:50:50 on dist-test-slave-65tz"
server_key: "73f1ad3dfd21fbb43b695194e883c73c"
server_key_iv: "b46e0fbe8d9992d355a0bb3db693579e"
server_key_version: "encryptionkey@0"
I20260620 14:50:50.816287 23045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "a9301e8263f34d22a8ad809ee4a2ed48"
format_stamp: "Formatted at 2026-06-20 14:50:50 on dist-test-slave-65tz"
server_key: "73f1ad3dfd21fbb43b695194e883c73c"
server_key_iv: "b46e0fbe8d9992d355a0bb3db693579e"
server_key_version: "encryptionkey@0"
I20260620 14:50:50.820750 23045 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 14:50:50.824368 23060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:50.826162 23045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:50.826316 23045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "a9301e8263f34d22a8ad809ee4a2ed48"
format_stamp: "Formatted at 2026-06-20 14:50:50 on dist-test-slave-65tz"
server_key: "73f1ad3dfd21fbb43b695194e883c73c"
server_key_iv: "b46e0fbe8d9992d355a0bb3db693579e"
server_key_version: "encryptionkey@0"
I20260620 14:50:50.826457 23045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:50.843101 23045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:50.847602 23045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:50.847844 23045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:50.848551 23045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:50.849545 23045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:50.849601 23045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:50.849681 23045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:50.849723 23045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:50.861240 23045 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:36261
I20260620 14:50:50.861280 23173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:36261 every 8 connection(s)
I20260620 14:50:50.862453 23045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:50:50.863241 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 23045
I20260620 14:50:50.863418 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:50:50.863726 9079 external_mini_cluster.cc:1428] Setting key 59db8717d70bd19e11437bbec2a9ed16
Jun 20 14:50:50 dist-test-slave-65tz krb5kdc[22804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967050, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.195@KRBTEST.COM" created.
I20260620 14:50:50.878607 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:50.864730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:42087 (local address 127.8.221.254:43407)
0620 14:50:50.864873 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:50.864878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:50.866112 (+ 1234us) server_negotiation.cc:407] Connection header received
0620 14:50:50.867502 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:50.867506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:50.867556 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:50.867634 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:50.869274 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:50.869858 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:50.871014 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:50.871258 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:50.871371 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:50.874469 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:50.874475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:50.874478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:50.874533 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:50.876079 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:50.876671 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:50.876674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:50.876676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:50.876821 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:50.877281 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:50.877284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:50.877285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:50.877506 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:50.877594 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:50.878328 (+ 734us) server_negotiation.cc:299] Negotiation successful
0620 14:50:50.878452 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:50:50.879685 23176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:50.865275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:42087)
0620 14:50:50.865956 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:50.866009 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:50.867289 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:50.868232 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:50.868239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:50.868610 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:50.869121 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:50.869134 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:50.869988 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:50.869996 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:50.870885 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:50.870895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:50.871418 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:50.871422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:50.871591 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:50.872245 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:50.872262 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:50.874193 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:50.876232 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:50.876264 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:50.876278 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:50.876554 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:50.876963 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:50.876966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:50.876969 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:50.877135 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:50.877625 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:50.877634 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:50.878032 (+ 398us) client_negotiation.cc:769] Sending connection context
0620 14:50:50.878276 (+ 244us) client_negotiation.cc:240] Negotiation successful
0620 14:50:50.878673 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":190,"threads_started":1}
I20260620 14:50:50.881157 23174 heartbeater.cc:347] Connected to a master server at 127.8.221.254:43407
I20260620 14:50:50.881417 23174 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:50.882059 23174 heartbeater.cc:511] Master 127.8.221.254:43407 requested a full tablet report, sending...
I20260620 14:50:50.883136 22849 ts_manager.cc:194] Registered new tserver with Master: a9301e8263f34d22a8ad809ee4a2ed48 (127.8.221.194:36261)
I20260620 14:50:50.883754 22849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:42087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
I20260620 14:50:50.895147 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:43407
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:51.034291 23179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:51.034601 23179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:51.034651 23179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:51.038906 23179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:51.038973 23179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:51.039032 23179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:50:51.044565 23179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:43407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:51.045825 23179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:51.046979 23179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:51.058485 23185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:51.058499 23184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:51.058511 23187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:51.058780 23179 server_base.cc:1034] running on GCE node
I20260620 14:50:51.059248 23179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:51.060101 23179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:51.061347 23179 hybrid_clock.cc:648] HybridClock initialized: now 1781967051061330 us; error 36 us; skew 500 ppm
Jun 20 14:50:51 dist-test-slave-65tz krb5kdc[22804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967051, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:51.064718 23179 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:50:51.066053 23179 webserver.cc:460] Webserver started at http://127.8.221.195:33143/ using document root <none> and password file <none>
I20260620 14:50:51.066823 23179 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:51.066898 23179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:51.067157 23179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:51.069774 23179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "db1e55097d214ce4bc7cceebf840fc36"
format_stamp: "Formatted at 2026-06-20 14:50:51 on dist-test-slave-65tz"
server_key: "1813c2bd82f8e70941934bd36a4c3904"
server_key_iv: "a5b39f1a2b663927aa166b46b7355302"
server_key_version: "encryptionkey@0"
I20260620 14:50:51.070549 23179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "db1e55097d214ce4bc7cceebf840fc36"
format_stamp: "Formatted at 2026-06-20 14:50:51 on dist-test-slave-65tz"
server_key: "1813c2bd82f8e70941934bd36a4c3904"
server_key_iv: "a5b39f1a2b663927aa166b46b7355302"
server_key_version: "encryptionkey@0"
I20260620 14:50:51.075762 23179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 14:50:51.079378 23194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:51.081059 23179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:50:51.081377 23179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "db1e55097d214ce4bc7cceebf840fc36"
format_stamp: "Formatted at 2026-06-20 14:50:51 on dist-test-slave-65tz"
server_key: "1813c2bd82f8e70941934bd36a4c3904"
server_key_iv: "a5b39f1a2b663927aa166b46b7355302"
server_key_version: "encryptionkey@0"
I20260620 14:50:51.081529 23179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:51.096004 23179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:51.099395 23179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:51.099599 23179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:51.100219 23179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:51.101086 23179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:51.101167 23179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:51.101239 23179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:51.101284 23179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:51.111243 23179 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:45859
I20260620 14:50:51.111267 23307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:45859 every 8 connection(s)
I20260620 14:50:51.112246 23179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:50:51.113914 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 23179
I20260620 14:50:51.114060 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:50:51.114450 9079 external_mini_cluster.cc:1428] Setting key 3239e897a8d2cd236bb961f94066132e
Jun 20 14:50:51 dist-test-slave-65tz krb5kdc[22804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967051, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:50:51.129077 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:51.115024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:56893 (local address 127.8.221.254:43407)
0620 14:50:51.115375 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:51.115379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:51.116273 (+ 894us) server_negotiation.cc:407] Connection header received
0620 14:50:51.118032 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:51.118035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:51.118091 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:51.118168 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:51.119505 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.120118 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.121482 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.121741 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.121905 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:51.124675 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:51.124678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:51.124680 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:51.124704 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:51.126318 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:51.126904 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:51.126907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:51.126908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:51.126980 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:51.127497 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:51.127501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:51.127502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:51.127850 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:51.127958 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:51.128735 (+ 777us) server_negotiation.cc:299] Negotiation successful
0620 14:50:51.128875 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256}
I20260620 14:50:51.130425 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:51.115471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.195:56893)
0620 14:50:51.116111 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:51.116164 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:51.117641 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:51.118340 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:51.118348 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:51.118771 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:51.119346 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:51.119360 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.120281 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.120285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:51.121349 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:51.121359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.121950 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.121953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:51.122124 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:51.122802 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:51.122820 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:51.124506 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:51.126455 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:51.126461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:51.126475 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:51.126759 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:51.127100 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:51.127103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:51.127105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:51.127337 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:51.127984 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:51.127992 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:51.128401 (+ 409us) client_negotiation.cc:769] Sending connection context
0620 14:50:51.128656 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 14:50:51.129091 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":161,"threads_started":1}
I20260620 14:50:51.131814 23308 heartbeater.cc:347] Connected to a master server at 127.8.221.254:43407
I20260620 14:50:51.132078 23308 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:51.132627 23308 heartbeater.cc:511] Master 127.8.221.254:43407 requested a full tablet report, sending...
I20260620 14:50:51.133841 22849 ts_manager.cc:194] Registered new tserver with Master: db1e55097d214ce4bc7cceebf840fc36 (127.8.221.195:45859)
I20260620 14:50:51.134460 22849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:56893
I20260620 14:50:51.141332 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:50:51.151008 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:51.143494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54790 (local address 127.8.221.254:43407)
0620 14:50:51.143628 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:51.143634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:51.143838 (+ 204us) server_negotiation.cc:407] Connection header received
0620 14:50:51.143995 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:51.143999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:51.144042 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:51.144144 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:51.144838 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.145382 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.146309 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.146537 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.146647 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:51.148032 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:51.148036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:51.148039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:51.148068 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:51.149311 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:51.149719 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:51.149724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:51.149726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:51.149852 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:51.150141 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:51.150145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:51.150146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:51.150369 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:51.150509 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:51.150733 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 14:50:51.150843 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:50:51.154135 22849 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:50:51.157269 22849 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:50:51.174998 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:51.169372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51474 (local address 127.8.221.195:45859)
0620 14:50:51.169851 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:51.169865 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:51.169898 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:50:51.170002 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:51.170008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:51.170242 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:51.170418 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:51.170846 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.171910 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.173088 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.173917 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.174025 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:51.174262 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:51.174723 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 14:50:51.174840 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":144,"threads_started":1}
I20260620 14:50:51.175190 23319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:51.168276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:36261 (local address 127.0.0.1:35148)
0620 14:50:51.169198 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:51.169214 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:51.169322 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:51.170271 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:51.170274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:51.170300 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:51.170460 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:51.170468 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.171709 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.171715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:51.173430 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:51.173441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.174650 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.174655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:51.174777 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:51.174808 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:50:51.174914 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:50:51.175010 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":150,"thread_start_us":157,"threads_started":1}
I20260620 14:50:51.175233 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:51.168944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35148 (local address 127.8.221.194:36261)
0620 14:50:51.169820 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:51.169833 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:51.169855 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:50:51.169929 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:51.169937 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:51.170109 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:51.170609 (+ 500us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:51.170765 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.171575 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.173592 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.174502 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.174645 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:51.174795 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:51.174958 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:50:51.175055 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":170,"threads_started":1}
I20260620 14:50:51.175546 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:51.168139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:40425 (local address 127.0.0.1:51706)
0620 14:50:51.168765 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:51.168818 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:51.168954 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:51.170032 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:51.170039 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:51.170063 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:51.170264 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:51.170272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.171875 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.171878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:51.173430 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:51.173441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.174650 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.174654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:51.175136 (+ 482us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:51.175156 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:50:51.175218 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:50:51.175413 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":115,"thread_start_us":114,"threads_started":1}
I20260620 14:50:51.174991 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:51.169185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:45859 (local address 127.0.0.1:51474)
0620 14:50:51.169515 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:51.169526 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:51.169638 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:51.170607 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:51.170610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:51.170626 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:51.170713 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:51.170719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.172049 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.172055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:51.172958 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:51.172966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.174326 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.174329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:51.174465 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:51.174516 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 14:50:51.174646 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 14:50:51.174802 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":9,"thread_start_us":115,"threads_started":1}
I20260620 14:50:51.176213 23321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:51.168315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.8.221.193:40425)
0620 14:50:51.169459 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:51.169470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:51.169502 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:50:51.169598 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:51.169603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:51.169864 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:51.170025 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:51.170438 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.171743 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.173593 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.174502 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.175747 (+ 1245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:51.175902 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:51.175958 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 14:50:51.176061 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1022,"thread_start_us":141,"threads_started":1}
I20260620 14:50:51.178984 23108 tablet_service.cc:1467] Processing CreateTablet for tablet def20c456a2649129f2ae70992b26b7a (DEFAULT_TABLE table=test-table [id=a4ef5ae72ede485c9117f3b68890ae3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:51.179196 23242 tablet_service.cc:1467] Processing CreateTablet for tablet def20c456a2649129f2ae70992b26b7a (DEFAULT_TABLE table=test-table [id=a4ef5ae72ede485c9117f3b68890ae3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:51.180053 23108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet def20c456a2649129f2ae70992b26b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:51.180410 23242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet def20c456a2649129f2ae70992b26b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:51.184329 22974 tablet_service.cc:1467] Processing CreateTablet for tablet def20c456a2649129f2ae70992b26b7a (DEFAULT_TABLE table=test-table [id=a4ef5ae72ede485c9117f3b68890ae3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:51.185598 22974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet def20c456a2649129f2ae70992b26b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:51.186923 23325 tablet_bootstrap.cc:492] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48: Bootstrap starting.
I20260620 14:50:51.188431 23326 tablet_bootstrap.cc:492] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36: Bootstrap starting.
I20260620 14:50:51.190483 23326 tablet_bootstrap.cc:654] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:51.190637 23325 tablet_bootstrap.cc:654] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:51.191568 23325 log.cc:824] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:51.191660 23326 log.cc:824] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:51.193895 23326 tablet_bootstrap.cc:492] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36: No bootstrap required, opened a new log
I20260620 14:50:51.193876 23329 tablet_bootstrap.cc:492] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869: Bootstrap starting.
I20260620 14:50:51.194044 23325 tablet_bootstrap.cc:492] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48: No bootstrap required, opened a new log
I20260620 14:50:51.194164 23326 ts_tablet_manager.cc:1397] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:50:51.194193 23325 ts_tablet_manager.cc:1397] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 14:50:51.196090 23329 tablet_bootstrap.cc:654] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:51.197043 23329 log.cc:824] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:51.198501 23325 raft_consensus.cc:348] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } }
I20260620 14:50:51.198419 23326 raft_consensus.cc:348] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } }
I20260620 14:50:51.198740 23326 raft_consensus.cc:374] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:51.198740 23325 raft_consensus.cc:374] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:51.198803 23326 raft_consensus.cc:729] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1e55097d214ce4bc7cceebf840fc36, State: Initialized, Role: FOLLOWER
I20260620 14:50:51.198803 23325 raft_consensus.cc:729] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9301e8263f34d22a8ad809ee4a2ed48, State: Initialized, Role: FOLLOWER
I20260620 14:50:51.199160 23329 tablet_bootstrap.cc:492] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869: No bootstrap required, opened a new log
I20260620 14:50:51.199290 23325 consensus_queue.cc:260] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } }
I20260620 14:50:51.199290 23326 consensus_queue.cc:260] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } }
I20260620 14:50:51.199389 23329 ts_tablet_manager.cc:1397] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:50:51.200062 23174 heartbeater.cc:503] Master 127.8.221.254:43407 was elected leader, sending a full tablet report...
I20260620 14:50:51.200358 23326 ts_tablet_manager.cc:1428] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:50:51.200491 23308 heartbeater.cc:503] Master 127.8.221.254:43407 was elected leader, sending a full tablet report...
I20260620 14:50:51.201357 23325 ts_tablet_manager.cc:1428] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 14:50:51.205235 23329 raft_consensus.cc:348] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } }
I20260620 14:50:51.205596 23329 raft_consensus.cc:374] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:51.205662 23329 raft_consensus.cc:729] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff3fa9f4ed64017aad4d7f2804c0869, State: Initialized, Role: FOLLOWER
I20260620 14:50:51.206260 23329 consensus_queue.cc:260] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } }
I20260620 14:50:51.207079 23040 heartbeater.cc:503] Master 127.8.221.254:43407 was elected leader, sending a full tablet report...
I20260620 14:50:51.207445 23329 ts_tablet_manager.cc:1428] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260620 14:50:51.363485 23309 tablet.cc:2369] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:50:51.363816 23175 tablet.cc:2369] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:50:51.376910 23041 tablet.cc:2369] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:50:51.403280 23331 raft_consensus.cc:482] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:50:51.403589 23331 raft_consensus.cc:504] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } }
I20260620 14:50:51.405063 23331 leader_election.cc:288] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff3fa9f4ed64017aad4d7f2804c0869 (127.8.221.193:40425), a9301e8263f34d22a8ad809ee4a2ed48 (127.8.221.194:36261)
I20260620 14:50:51.409888 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:51.405376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:40425 (local address 127.8.221.195:58253)
0620 14:50:51.405539 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:51.405559 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:51.405668 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:51.406102 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:51.406104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:51.406120 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:51.406283 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:51.406291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.407324 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.407328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:51.408359 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:51.408371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.409359 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.409363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:51.409483 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:51.409514 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:50:51.409598 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:50:51.409685 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 14:50:51.409868 23321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:51.405527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:58253 (local address 127.8.221.193:40425)
0620 14:50:51.405729 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:51.405734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:51.405770 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:50:51.405890 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:51.405893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:51.405978 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:51.406081 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:51.406428 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.407211 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.408511 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.409230 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.409338 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:51.409385 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:51.409625 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 14:50:51.409689 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:50:51.410670 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:51.405395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:36261 (local address 127.8.221.195:46979)
0620 14:50:51.406168 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:51.406188 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:51.406312 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:51.406621 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:51.406624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:51.406643 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:51.406789 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:51.406796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.407747 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.407751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:51.408952 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:51.409012 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.409928 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.409935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:51.410246 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:51.410286 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:50:51.410413 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 14:50:51.410497 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":210,"thread_start_us":262,"threads_started":1}
I20260620 14:50:51.410707 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:51.405494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:46979 (local address 127.8.221.194:36261)
0620 14:50:51.405635 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:51.405642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:51.406297 (+ 655us) server_negotiation.cc:407] Connection header received
0620 14:50:51.406444 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:51.406447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:51.406500 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:51.406569 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:51.406907 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.407620 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.409139 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.409735 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.409909 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:51.409977 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:51.410449 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 14:50:51.410586 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:50:51.410657 22994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def20c456a2649129f2ae70992b26b7a" candidate_uuid: "db1e55097d214ce4bc7cceebf840fc36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" is_pre_election: true
I20260620 14:50:51.411124 22994 raft_consensus.cc:2454] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1e55097d214ce4bc7cceebf840fc36 in term 0.
I20260620 14:50:51.411520 23128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def20c456a2649129f2ae70992b26b7a" candidate_uuid: "db1e55097d214ce4bc7cceebf840fc36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" is_pre_election: true
I20260620 14:50:51.411732 23195 leader_election.cc:302] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aff3fa9f4ed64017aad4d7f2804c0869, db1e55097d214ce4bc7cceebf840fc36; no voters:
I20260620 14:50:51.411890 23128 raft_consensus.cc:2454] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db1e55097d214ce4bc7cceebf840fc36 in term 0.
I20260620 14:50:51.412119 23331 raft_consensus.cc:2781] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:50:51.412247 23331 raft_consensus.cc:482] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:50:51.412321 23331 raft_consensus.cc:3037] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:51.413559 23331 raft_consensus.cc:504] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } }
I20260620 14:50:51.414021 23331 leader_election.cc:288] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [CANDIDATE]: Term 1 election: Requested vote from peers aff3fa9f4ed64017aad4d7f2804c0869 (127.8.221.193:40425), a9301e8263f34d22a8ad809ee4a2ed48 (127.8.221.194:36261)
I20260620 14:50:51.414403 23128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def20c456a2649129f2ae70992b26b7a" candidate_uuid: "db1e55097d214ce4bc7cceebf840fc36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9301e8263f34d22a8ad809ee4a2ed48"
I20260620 14:50:51.414444 22994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "def20c456a2649129f2ae70992b26b7a" candidate_uuid: "db1e55097d214ce4bc7cceebf840fc36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff3fa9f4ed64017aad4d7f2804c0869"
I20260620 14:50:51.414522 23128 raft_consensus.cc:3037] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:51.414644 22994 raft_consensus.cc:3037] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:51.415655 23128 raft_consensus.cc:2454] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1e55097d214ce4bc7cceebf840fc36 in term 1.
I20260620 14:50:51.416011 23198 leader_election.cc:302] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9301e8263f34d22a8ad809ee4a2ed48, db1e55097d214ce4bc7cceebf840fc36; no voters:
I20260620 14:50:51.416167 22994 raft_consensus.cc:2454] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db1e55097d214ce4bc7cceebf840fc36 in term 1.
I20260620 14:50:51.416260 23331 raft_consensus.cc:2781] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:51.416527 23331 raft_consensus.cc:686] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [term 1 LEADER]: Becoming Leader. State: Replica: db1e55097d214ce4bc7cceebf840fc36, State: Running, Role: LEADER
I20260620 14:50:51.416936 23331 consensus_queue.cc:237] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } }
I20260620 14:50:51.421265 22847 catalog_manager.cc:5495] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 reported cstate change: term changed from 0 to 1, leader changed from <none> to db1e55097d214ce4bc7cceebf840fc36 (127.8.221.195). New cstate: current_term: 1 leader_uuid: "db1e55097d214ce4bc7cceebf840fc36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } health_report { overall_health: HEALTHY } } }
I20260620 14:50:51.469749 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:51.465888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51480 (local address 127.8.221.195:45859)
0620 14:50:51.466053 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:51.466058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:51.466073 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:50:51.466162 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:51.466164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:51.466226 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:51.466340 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:51.466633 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.467221 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.468149 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:51.468381 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:51.468480 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:51.468800 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:51.469328 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:51.469431 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:51.469583 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 14:50:51.469635 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:50:51.476380 22994 raft_consensus.cc:1261] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Refusing update from remote peer db1e55097d214ce4bc7cceebf840fc36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:51.476449 23128 raft_consensus.cc:1261] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48 [term 1 FOLLOWER]: Refusing update from remote peer db1e55097d214ce4bc7cceebf840fc36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:51.477121 23331 consensus_queue.cc:1035] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:51.477344 23335 consensus_queue.cc:1035] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:51.487501 23340 mvcc.cc:205] Tried to move back new op lower bound from 7298937042839064576 to 7298937042606743552. Current Snapshot: MvccSnapshot[applied={T|T < 7298937042839064576}]
I20260620 14:50:51.497430 22847 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:51.497596 22847 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:51.500393 22847 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb: Sending DeleteTablet for 3 replicas of tablet def20c456a2649129f2ae70992b26b7a
I20260620 14:50:51.501329 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 22911
I20260620 14:50:51.501492 23242 tablet_service.cc:1514] Processing DeleteTablet for tablet def20c456a2649129f2ae70992b26b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:51 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:51474
I20260620 14:50:51.501495 23108 tablet_service.cc:1514] Processing DeleteTablet for tablet def20c456a2649129f2ae70992b26b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:51 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:35148
I20260620 14:50:51.502588 23348 tablet_replica.cc:330] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36: stopping tablet replica
I20260620 14:50:51.502910 23348 raft_consensus.cc:2229] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:50:51.503360 23348 raft_consensus.cc:2258] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:51.503402 23349 tablet_replica.cc:330] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48: stopping tablet replica
I20260620 14:50:51.503931 23349 raft_consensus.cc:2229] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:51.504405 23349 raft_consensus.cc:2258] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:51.504966 23348 ts_tablet_manager.cc:1905] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:51.506409 23349 ts_tablet_manager.cc:1905] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:51.508589 23348 ts_tablet_manager.cc:1918] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:51.508713 23348 log.cc:1196] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/def20c456a2649129f2ae70992b26b7a
I20260620 14:50:51.509096 23348 ts_tablet_manager.cc:1939] T def20c456a2649129f2ae70992b26b7a P db1e55097d214ce4bc7cceebf840fc36: Deleting consensus metadata
I20260620 14:50:51.509974 23349 ts_tablet_manager.cc:1918] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260620 14:50:51.510015 22833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.221.193:40425 (error 108)
I20260620 14:50:51.510180 23349 log.cc:1196] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/def20c456a2649129f2ae70992b26b7a
I20260620 14:50:51.510497 22833 catalog_manager.cc:4841] TS db1e55097d214ce4bc7cceebf840fc36 (127.8.221.195:45859): tablet def20c456a2649129f2ae70992b26b7a (table test-table [id=a4ef5ae72ede485c9117f3b68890ae3e]) successfully deleted
I20260620 14:50:51.510655 23349 ts_tablet_manager.cc:1939] T def20c456a2649129f2ae70992b26b7a P a9301e8263f34d22a8ad809ee4a2ed48: Deleting consensus metadata
I20260620 14:50:51.511061 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 23045
I20260620 14:50:51.511119 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:51.510794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:40425 (local address 127.0.0.1:51716)
0620 14:50:51.510971 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:51.511037 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.193:40425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260620 14:50:51.513098 22833 catalog_manager.cc:4584] TS aff3fa9f4ed64017aad4d7f2804c0869 (127.8.221.193:40425): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet def20c456a2649129f2ae70992b26b7a: Network error: Client connection negotiation failed: client connection to 127.8.221.193:40425: connect: Connection refused (error 111)
I20260620 14:50:51.517493 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 23179
I20260620 14:50:51.519305 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:51.518980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:36261 (local address 127.0.0.1:35164)
0620 14:50:51.519154 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:51.519213 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.194:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 14:50:51.524308 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 22817
I20260620 14:50:51.532510 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:43407
--webserver_interface=127.8.221.254
--webserver_port=41951
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:43407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:51.660662 23351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:51.661008 23351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:51.661052 23351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:51.665517 23351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:51.665617 23351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:51.665638 23351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:51.665656 23351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:51.665692 23351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:51.671602 23351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:43407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:43407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=41951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:51.672974 23351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:51.674286 23351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:51.690254 23357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:51.690320 23358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:51.690272 23360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:51.690819 23351 server_base.cc:1034] running on GCE node
I20260620 14:50:51.691653 23351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:51.693177 23351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:51.694377 23351 hybrid_clock.cc:648] HybridClock initialized: now 1781967051694354 us; error 47 us; skew 500 ppm
Jun 20 14:50:51 dist-test-slave-65tz krb5kdc[22804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967051, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:51.698096 23351 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:50:51.699393 23351 webserver.cc:460] Webserver started at http://127.8.221.254:41951/ using document root <none> and password file <none>
I20260620 14:50:51.700217 23351 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:51.700347 23351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:51.705642 23351 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:50:51.708637 23367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:51.710326 23351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:51.710501 23351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "1621fd4489704ef9a2f1f48d87190fbb"
format_stamp: "Formatted at 2026-06-20 14:50:50 on dist-test-slave-65tz"
server_key: "0deb52d1f9e8b26ae72a632ba2444dad"
server_key_iv: "b08d1301e48649c1c4f5bd3d575207f1"
server_key_version: "encryptionkey@0"
I20260620 14:50:51.711285 23351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:51.745529 23351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:51.749825 23351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:51.750069 23351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:51.759284 23351 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:43407
I20260620 14:50:51.759606 23419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:43407 every 8 connection(s)
I20260620 14:50:51.760849 23351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:51.765774 23420 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:50:51.767657 23420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb: Bootstrap starting.
I20260620 14:50:51.771212 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 23351
I20260620 14:50:51.771781 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:40425
--local_ip_for_outbound_sockets=127.8.221.193
--tserver_master_addrs=127.8.221.254:43407
--webserver_port=34955
--webserver_interface=127.8.221.193
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:50:51.785365 23420 log.cc:824] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:51.794272 23420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:50:51.794694 23420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb: Bootstrap complete.
I20260620 14:50:51.799187 23420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1621fd4489704ef9a2f1f48d87190fbb" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43407 } }
I20260620 14:50:51.799635 23420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1621fd4489704ef9a2f1f48d87190fbb, State: Initialized, Role: FOLLOWER
I20260620 14:50:51.800201 23420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1621fd4489704ef9a2f1f48d87190fbb" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43407 } }
I20260620 14:50:51.800352 23420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:51.800444 23420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:51.800563 23420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:50:51.802726 23420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1621fd4489704ef9a2f1f48d87190fbb" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43407 } }
I20260620 14:50:51.803097 23420 leader_election.cc:302] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1621fd4489704ef9a2f1f48d87190fbb; no voters:
I20260620 14:50:51.803443 23420 leader_election.cc:288] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:50:51.803683 23424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:50:51.804206 23424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 2 LEADER]: Becoming Leader. State: Replica: 1621fd4489704ef9a2f1f48d87190fbb, State: Running, Role: LEADER
I20260620 14:50:51.804723 23424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1621fd4489704ef9a2f1f48d87190fbb" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43407 } }
I20260620 14:50:51.804797 23420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:51.806579 23426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1621fd4489704ef9a2f1f48d87190fbb. Latest consensus state: current_term: 2 leader_uuid: "1621fd4489704ef9a2f1f48d87190fbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1621fd4489704ef9a2f1f48d87190fbb" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43407 } } }
I20260620 14:50:51.806769 23426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:51.807140 23425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1621fd4489704ef9a2f1f48d87190fbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1621fd4489704ef9a2f1f48d87190fbb" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43407 } } }
I20260620 14:50:51.807260 23425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:51.811872 23439 catalog_manager.cc:1259] Loaded cluster ID: 43eae1cb22eb4f96a1618abe483e3adf
I20260620 14:50:51.811949 23439 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb: loading cluster ID for follower catalog manager: success
W20260620 14:50:51.813185 23439 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260620 14:50:51.813268 23439 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260620 14:50:51.813390 23438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:51.816514 23438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:51.817073 23438 catalog_manager.cc:1259] Loaded cluster ID: 43eae1cb22eb4f96a1618abe483e3adf
I20260620 14:50:51.817145 23438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:51.818472 23438 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 14:50:51.823396 23438 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 14:50:51.830660 23438 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:51.831926 23438 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb: Loaded TSK: 0
I20260620 14:50:51.833691 23438 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:50:51.915421 23422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:51.915714 23422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:51.915768 23422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:51.920149 23422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:51.920217 23422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:51.920326 23422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:50:51.926051 23422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:40425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=34955
--tserver_master_addrs=127.8.221.254:43407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:51.927434 23422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:51.928726 23422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:51.942107 23450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:51.942056 23447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:51.942025 23448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:51.942803 23422 server_base.cc:1034] running on GCE node
I20260620 14:50:51.943454 23422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:51.944412 23422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:51.945676 23422 hybrid_clock.cc:648] HybridClock initialized: now 1781967051945638 us; error 60 us; skew 500 ppm
Jun 20 14:50:51 dist-test-slave-65tz krb5kdc[22804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967051, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:51.949254 23422 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:50:51.950529 23422 webserver.cc:460] Webserver started at http://127.8.221.193:34955/ using document root <none> and password file <none>
I20260620 14:50:51.951227 23422 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:51.951339 23422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:51.956240 23422 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:50:51.959342 23457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:51.961046 23422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:50:51.961232 23422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "aff3fa9f4ed64017aad4d7f2804c0869"
format_stamp: "Formatted at 2026-06-20 14:50:50 on dist-test-slave-65tz"
server_key: "f1083f14ef0729e5b606d9c8d91f9c95"
server_key_iv: "7db513b06b2198e38e54e8c4f08c3007"
server_key_version: "encryptionkey@0"
I20260620 14:50:51.961974 23422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:51.984935 23422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:51.989094 23422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:51.989358 23422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:51.990236 23422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:51.991686 23464 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:50:51.996336 23422 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:50:51.996438 23422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260620 14:50:51.996546 23422 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:50:51.999023 23422 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:50:51.999104 23422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260620 14:50:51.999464 23464 tablet_bootstrap.cc:492] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869: Bootstrap starting.
I20260620 14:50:52.012004 23422 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:40425
I20260620 14:50:52.012142 23571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:40425 every 8 connection(s)
I20260620 14:50:52.013269 23422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:50:52.019068 23464 log.cc:824] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:52.021347 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 23422
I20260620 14:50:52.021960 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:36261
--local_ip_for_outbound_sockets=127.8.221.194
--tserver_master_addrs=127.8.221.254:43407
--webserver_port=37053
--webserver_interface=127.8.221.194
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:50:52 dist-test-slave-65tz krb5kdc[22804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967051, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:50:52.028362 23464 tablet_bootstrap.cc:492] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:50:52.028788 23464 tablet_bootstrap.cc:492] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869: Bootstrap complete.
I20260620 14:50:52.030326 23464 ts_tablet_manager.cc:1397] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.008s
I20260620 14:50:52.033443 23464 raft_consensus.cc:348] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } }
I20260620 14:50:52.033960 23464 raft_consensus.cc:729] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff3fa9f4ed64017aad4d7f2804c0869, State: Initialized, Role: FOLLOWER
I20260620 14:50:52.034500 23464 consensus_queue.cc:260] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } }
I20260620 14:50:52.035668 23464 ts_tablet_manager.cc:1428] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 14:50:52.036628 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.016049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:40043 (local address 127.8.221.254:43407)
0620 14:50:52.016514 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:52.016532 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:52.017741 (+ 1209us) server_negotiation.cc:407] Connection header received
0620 14:50:52.019136 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:52.019196 (+ 60us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:52.019670 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:52.020126 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:52.021658 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.023397 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.025146 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.025703 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.025950 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:52.029501 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:52.029513 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:52.029536 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:52.029587 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:52.032135 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:52.033087 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:52.033093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:52.033099 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:52.033276 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:52.033894 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:52.033897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:52.033899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:52.034403 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:52.034554 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:52.035505 (+ 951us) server_negotiation.cc:299] Negotiation successful
0620 14:50:52.035866 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":152,"threads_started":1}
I20260620 14:50:52.037048 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.016054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:40043)
0620 14:50:52.017511 (+ 1457us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:52.017574 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:52.018654 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:52.020201 (+ 1547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:52.020209 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:52.020766 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:52.021437 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:52.021458 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.023613 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.023616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:52.024978 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:52.024991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.025899 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.025903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:52.026094 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:52.026895 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:52.026916 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:52.029094 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:52.032331 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:52.032343 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:52.032367 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:52.032922 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:52.033412 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:52.033418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:52.033422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:52.033739 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:52.034563 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:52.034573 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:52.035051 (+ 478us) client_negotiation.cc:769] Sending connection context
0620 14:50:52.035327 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 14:50:52.035861 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1192,"thread_start_us":205,"threads_started":1}
I20260620 14:50:52.040038 23572 heartbeater.cc:347] Connected to a master server at 127.8.221.254:43407
I20260620 14:50:52.040350 23572 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:52.040934 23572 heartbeater.cc:511] Master 127.8.221.254:43407 requested a full tablet report, sending...
I20260620 14:50:52.044296 23384 ts_manager.cc:194] Registered new tserver with Master: aff3fa9f4ed64017aad4d7f2804c0869 (127.8.221.193:40425)
I20260620 14:50:52.049947 23384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:40043
I20260620 14:50:52.052747 23572 heartbeater.cc:503] Master 127.8.221.254:43407 was elected leader, sending a full tablet report...
Jun 20 14:50:52 dist-test-slave-65tz krb5kdc[22804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967051, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for kudu/127.8.221.193@KRBTEST.COM
I20260620 14:50:52.062337 23580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.048675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.8.221.193:40425)
0620 14:50:52.050023 (+ 1348us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:52.050039 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:52.050068 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:50:52.050157 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:52.050163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:52.050404 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:52.050570 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:52.051556 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.052768 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.054083 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.054575 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.055214 (+ 639us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:52.057741 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:52.057751 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:52.057764 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:52.057942 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:52.060028 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:52.060664 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:52.060670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:52.060671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:52.060759 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:52.061126 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:52.061129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:52.061130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:52.061433 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:52.061564 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:52.062035 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 14:50:52.062199 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1200,"thread_start_us":168,"threads_started":1}
I20260620 14:50:52.062386 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.048505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:40425 (local address 127.0.0.1:51724)
0620 14:50:52.049380 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:52.049441 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:52.049620 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:52.050613 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:52.050620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:52.051201 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:52.051375 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:52.051385 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.052916 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.052920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:52.053927 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:52.053938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.054748 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.054752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:52.054864 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:52.055629 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:52.055647 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:52.057544 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:52.060174 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:52.060184 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:52.060190 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:52.060517 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:52.060867 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:52.060870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:52.060872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:52.060980 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:52.061560 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:52.061566 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:52.061734 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 14:50:52.061920 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 14:50:52.062170 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":182,"threads_started":1}
I20260620 14:50:52.063675 23506 tablet_service.cc:1514] Processing DeleteTablet for tablet def20c456a2649129f2ae70992b26b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:51 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:51724
I20260620 14:50:52.064546 23581 tablet_replica.cc:330] stopping tablet replica
I20260620 14:50:52.064862 23581 raft_consensus.cc:2229] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:52.065127 23581 raft_consensus.cc:2258] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:52.066298 23581 ts_tablet_manager.cc:1905] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:52.070024 23581 ts_tablet_manager.cc:1918] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:52.070148 23581 log.cc:1196] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/wal/wals/def20c456a2649129f2ae70992b26b7a
I20260620 14:50:52.070432 23581 ts_tablet_manager.cc:1939] T def20c456a2649129f2ae70992b26b7a P aff3fa9f4ed64017aad4d7f2804c0869: Deleting consensus metadata
I20260620 14:50:52.072265 23368 catalog_manager.cc:4841] TS aff3fa9f4ed64017aad4d7f2804c0869 (127.8.221.193:40425): tablet def20c456a2649129f2ae70992b26b7a (table test-table [id=a4ef5ae72ede485c9117f3b68890ae3e]) successfully deleted
W20260620 14:50:52.166374 23577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:52.166733 23577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:52.166785 23577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:52.171671 23577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:52.171757 23577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:52.171828 23577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:50:52.177775 23577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:36261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=37053
--tserver_master_addrs=127.8.221.254:43407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:52.179155 23577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:52.180361 23577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:52.192461 23589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:52.192618 23577 server_base.cc:1034] running on GCE node
W20260620 14:50:52.192480 23586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:52.192474 23587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:52.193454 23577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:52.194444 23577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:52.195645 23577 hybrid_clock.cc:648] HybridClock initialized: now 1781967052195620 us; error 39 us; skew 500 ppm
Jun 20 14:50:52 dist-test-slave-65tz krb5kdc[22804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967052, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:52.199164 23577 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:50:52.200400 23577 webserver.cc:460] Webserver started at http://127.8.221.194:37053/ using document root <none> and password file <none>
I20260620 14:50:52.201262 23577 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:52.201383 23577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:52.206738 23577 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:50:52.209381 23596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:52.211053 23577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:52.211247 23577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "a9301e8263f34d22a8ad809ee4a2ed48"
format_stamp: "Formatted at 2026-06-20 14:50:50 on dist-test-slave-65tz"
server_key: "73f1ad3dfd21fbb43b695194e883c73c"
server_key_iv: "b46e0fbe8d9992d355a0bb3db693579e"
server_key_version: "encryptionkey@0"
I20260620 14:50:52.212040 23577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:52.237275 23577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:52.241536 23577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:52.241907 23577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:52.242774 23577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:52.243866 23577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:52.243922 23577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:52.243973 23577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:52.243989 23577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:52.256595 23577 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:36261
I20260620 14:50:52.256635 23709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:36261 every 8 connection(s)
I20260620 14:50:52.257952 23577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:50:52.262598 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 23577
I20260620 14:50:52.263144 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:45859
--local_ip_for_outbound_sockets=127.8.221.195
--tserver_master_addrs=127.8.221.254:43407
--webserver_port=33143
--webserver_interface=127.8.221.195
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:50:52 dist-test-slave-65tz krb5kdc[22804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967052, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:50:52.277014 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.260530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:33701 (local address 127.8.221.254:43407)
0620 14:50:52.260692 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:52.260695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:52.262067 (+ 1372us) server_negotiation.cc:407] Connection header received
0620 14:50:52.263629 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:52.263633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:52.263693 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:52.263807 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:52.265514 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.266488 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.268424 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.268700 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.268807 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:52.272377 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:52.272381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:52.272383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:52.272415 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:52.274376 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:52.275110 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:52.275114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:52.275116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:52.275215 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:52.275701 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:52.275704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:52.275705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:52.275940 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:52.276038 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:52.276674 (+ 636us) server_negotiation.cc:299] Negotiation successful
0620 14:50:52.276857 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:50:52.278173 23712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.261149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:33701)
0620 14:50:52.261849 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:52.261921 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:52.263377 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:52.264006 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:52.264013 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:52.264501 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:52.265289 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:52.265310 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.266654 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.266661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:52.268269 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:52.268284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.268891 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.268897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:52.269145 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:52.269937 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:52.269964 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:52.272083 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:52.274515 (+ 2432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:52.274526 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:52.274545 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:52.274972 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:52.275331 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:52.275337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:52.275341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:52.275565 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:52.276061 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:52.276067 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:52.276399 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 14:50:52.276629 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 14:50:52.276980 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":172,"threads_started":1}
I20260620 14:50:52.279803 23710 heartbeater.cc:347] Connected to a master server at 127.8.221.254:43407
I20260620 14:50:52.280084 23710 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:52.280682 23710 heartbeater.cc:511] Master 127.8.221.254:43407 requested a full tablet report, sending...
I20260620 14:50:52.282105 23384 ts_manager.cc:194] Registered new tserver with Master: a9301e8263f34d22a8ad809ee4a2ed48 (127.8.221.194:36261)
I20260620 14:50:52.282861 23384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:33701
W20260620 14:50:52.386368 23713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:52.386648 23713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:52.386693 23713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:52.391054 23713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:52.391126 23713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:52.391189 23713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:50:52.396530 23713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:45859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=33143
--tserver_master_addrs=127.8.221.254:43407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:52.397619 23713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:52.398769 23713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:52.409154 23721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:52.409161 23719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:52.409162 23718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:52.409639 23713 server_base.cc:1034] running on GCE node
I20260620 14:50:52.410120 23713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:52.410835 23713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:52.412055 23713 hybrid_clock.cc:648] HybridClock initialized: now 1781967052412031 us; error 39 us; skew 500 ppm
Jun 20 14:50:52 dist-test-slave-65tz krb5kdc[22804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967052, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:52.415259 23713 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:50:52.416344 23713 webserver.cc:460] Webserver started at http://127.8.221.195:33143/ using document root <none> and password file <none>
I20260620 14:50:52.416986 23713 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:52.417097 23713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:52.421452 23713 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:50:52.424082 23728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:52.425735 23713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:50:52.425953 23713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "db1e55097d214ce4bc7cceebf840fc36"
format_stamp: "Formatted at 2026-06-20 14:50:51 on dist-test-slave-65tz"
server_key: "1813c2bd82f8e70941934bd36a4c3904"
server_key_iv: "a5b39f1a2b663927aa166b46b7355302"
server_key_version: "encryptionkey@0"
I20260620 14:50:52.426582 23713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:52.458719 23713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:52.462093 23713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:52.462316 23713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:52.462978 23713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:52.463896 23713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:52.463985 23713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:52.464088 23713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:52.464146 23713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:52.475939 23713 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:45859
I20260620 14:50:52.476469 23841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:45859 every 8 connection(s)
I20260620 14:50:52.477141 23713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:50:52.482084 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 23713
Jun 20 14:50:52 dist-test-slave-65tz krb5kdc[22804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967052, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:50:52.494415 23846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.482862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54804 (local address 127.8.221.254:43407)
0620 14:50:52.483159 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:52.483163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:52.483302 (+ 139us) server_negotiation.cc:407] Connection header received
0620 14:50:52.483464 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:52.483466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:52.483516 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:52.483614 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:52.484406 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.484993 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.486219 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.486553 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.486653 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:52.489230 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:52.489233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:52.489236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:52.489261 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:52.492137 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:52.492680 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:52.492683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:52.492684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:52.492811 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:52.493189 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:52.493191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:52.493192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:52.493483 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:52.493768 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:52.494009 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 14:50:52.494206 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":85,"threads_started":1}
I20260620 14:50:52.494729 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.479470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:52323 (local address 127.8.221.254:43407)
0620 14:50:52.479657 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:52.479661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:52.480692 (+ 1031us) server_negotiation.cc:407] Connection header received
0620 14:50:52.481845 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:52.481849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:52.481919 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:52.482033 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:52.483732 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.484371 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.486017 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.486330 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.486463 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:52.489854 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:52.489860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:52.489865 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:52.489917 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:52.492156 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:52.492804 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:52.492808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:52.492810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:52.492941 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:52.493359 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:52.493363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:52.493364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:52.493539 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:52.493636 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:52.494393 (+ 757us) server_negotiation.cc:299] Negotiation successful
0620 14:50:52.494542 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:50:52.495673 23844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.479870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.195:52323)
0620 14:50:52.480459 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:52.480511 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:52.481519 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:52.482267 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:52.482278 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:52.482798 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:52.483546 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:52.483563 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.484514 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.484518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:52.485883 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:52.485895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.486492 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.486495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:52.486698 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:52.487525 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:52.487548 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:52.489580 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:52.492282 (+ 2702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:52.492288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:52.492305 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:52.492648 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:52.493064 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:52.493067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:52.493069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:52.493256 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:52.493694 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:52.493704 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:52.494073 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 14:50:52.494325 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 14:50:52.494697 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":153,"threads_started":1}
I20260620 14:50:52.497112 23842 heartbeater.cc:347] Connected to a master server at 127.8.221.254:43407
I20260620 14:50:52.497421 23842 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:52.498056 23842 heartbeater.cc:511] Master 127.8.221.254:43407 requested a full tablet report, sending...
I20260620 14:50:52.499194 23384 ts_manager.cc:194] Registered new tserver with Master: db1e55097d214ce4bc7cceebf840fc36 (127.8.221.195:45859)
I20260620 14:50:52.499800 23384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:52323
I20260620 14:50:52.508734 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:50:52.513535 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.509490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.8.221.254:43407)
0620 14:50:52.509648 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:52.509652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:52.509677 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:50:52.509923 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:52.509926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:52.510070 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:52.510207 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:52.510453 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.511108 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.511975 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.512221 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.512311 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:52.512600 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:52.513165 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:52.513276 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:52.513330 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 14:50:52.513389 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:50:52.514429 23384 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:50:52.518332 23384 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:50:52.528570 23506 tablet_service.cc:1467] Processing CreateTablet for tablet 3c108d3047ba4f7c9260c480ee693dae (DEFAULT_TABLE table=test-table [id=d7282edd79d7407ab8351851d5ab1cae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:52.529402 23506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c108d3047ba4f7c9260c480ee693dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:52.532877 23464 tablet_bootstrap.cc:492] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869: Bootstrap starting.
I20260620 14:50:52.533499 23851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.527032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:36261 (local address 127.0.0.1:35174)
0620 14:50:52.527337 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:52.527357 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:52.527452 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:52.528630 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:52.528634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:52.528653 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:52.528789 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:52.528798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.530142 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.530145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:52.531752 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:52.531766 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.532804 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.532808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:52.532905 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:52.533070 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 14:50:52.533192 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 14:50:52.533280 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":90,"thread_start_us":133,"threads_started":1}
I20260620 14:50:52.533743 23852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.526958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51490 (local address 127.8.221.195:45859)
0620 14:50:52.527555 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:52.527571 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:52.527600 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:50:52.527700 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:52.527706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:52.527961 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:52.528197 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:52.528517 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.529932 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.531983 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.533038 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.533153 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:52.533317 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:52.533465 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 14:50:52.533577 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":193,"threads_started":1}
I20260620 14:50:52.533967 23464 tablet_bootstrap.cc:654] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:52.534197 23853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.527227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35174 (local address 127.8.221.194:36261)
0620 14:50:52.527959 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:52.527974 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:52.527999 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:50:52.528098 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:52.528105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:52.528283 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:52.528434 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:52.528943 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.529982 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.531904 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.532660 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.532803 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:52.532989 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:52.533916 (+ 927us) server_negotiation.cc:299] Negotiation successful
0620 14:50:52.534046 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":144,"threads_started":1}
I20260620 14:50:52.533690 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.526716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:45859 (local address 127.0.0.1:51490)
0620 14:50:52.526864 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:52.526887 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:52.526988 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:52.528196 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:52.528199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:52.528218 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:52.528350 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:52.528358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.530078 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.530085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:52.531830 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:52.531844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.533189 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.533192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:52.533308 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:52.533329 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:50:52.533414 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:50:52.533489 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":156}
I20260620 14:50:52.535709 23464 tablet_bootstrap.cc:492] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869: No bootstrap required, opened a new log
I20260620 14:50:52.535817 23464 ts_tablet_manager.cc:1397] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 14:50:52.536348 23464 raft_consensus.cc:348] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } }
I20260620 14:50:52.536479 23464 raft_consensus.cc:374] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:52.536512 23464 raft_consensus.cc:729] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff3fa9f4ed64017aad4d7f2804c0869, State: Initialized, Role: FOLLOWER
I20260620 14:50:52.536661 23464 consensus_queue.cc:260] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } }
I20260620 14:50:52.537016 23464 ts_tablet_manager.cc:1428] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260620 14:50:52.537943 23644 tablet_service.cc:1467] Processing CreateTablet for tablet 3c108d3047ba4f7c9260c480ee693dae (DEFAULT_TABLE table=test-table [id=d7282edd79d7407ab8351851d5ab1cae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:52.537948 23776 tablet_service.cc:1467] Processing CreateTablet for tablet 3c108d3047ba4f7c9260c480ee693dae (DEFAULT_TABLE table=test-table [id=d7282edd79d7407ab8351851d5ab1cae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:52.539171 23644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c108d3047ba4f7c9260c480ee693dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:52.539176 23776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c108d3047ba4f7c9260c480ee693dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:52.548204 23857 tablet_bootstrap.cc:492] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48: Bootstrap starting.
I20260620 14:50:52.548844 23856 tablet_bootstrap.cc:492] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36: Bootstrap starting.
I20260620 14:50:52.552050 23857 tablet_bootstrap.cc:654] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:52.552050 23856 tablet_bootstrap.cc:654] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:52.553121 23856 log.cc:824] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:52.553121 23857 log.cc:824] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:52.555578 23857 tablet_bootstrap.cc:492] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48: No bootstrap required, opened a new log
I20260620 14:50:52.555598 23856 tablet_bootstrap.cc:492] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36: No bootstrap required, opened a new log
I20260620 14:50:52.555827 23857 ts_tablet_manager.cc:1397] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 14:50:52.555851 23856 ts_tablet_manager.cc:1397] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 14:50:52.559396 23856 raft_consensus.cc:348] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } }
I20260620 14:50:52.559480 23857 raft_consensus.cc:348] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } }
I20260620 14:50:52.559685 23856 raft_consensus.cc:374] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:52.559767 23857 raft_consensus.cc:374] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:52.559772 23856 raft_consensus.cc:729] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db1e55097d214ce4bc7cceebf840fc36, State: Initialized, Role: FOLLOWER
I20260620 14:50:52.559841 23857 raft_consensus.cc:729] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9301e8263f34d22a8ad809ee4a2ed48, State: Initialized, Role: FOLLOWER
I20260620 14:50:52.560268 23856 consensus_queue.cc:260] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } }
I20260620 14:50:52.560365 23857 consensus_queue.cc:260] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } }
I20260620 14:50:52.561116 23842 heartbeater.cc:503] Master 127.8.221.254:43407 was elected leader, sending a full tablet report...
I20260620 14:50:52.561192 23710 heartbeater.cc:503] Master 127.8.221.254:43407 was elected leader, sending a full tablet report...
I20260620 14:50:52.561414 23856 ts_tablet_manager.cc:1428] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:50:52.561545 23857 ts_tablet_manager.cc:1428] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:50:52.578711 23861 raft_consensus.cc:482] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:50:52.578929 23861 raft_consensus.cc:504] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } }
I20260620 14:50:52.580124 23861 leader_election.cc:288] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db1e55097d214ce4bc7cceebf840fc36 (127.8.221.195:45859), aff3fa9f4ed64017aad4d7f2804c0869 (127.8.221.193:40425)
I20260620 14:50:52.584671 23712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.580399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:45859 (local address 127.8.221.194:60847)
0620 14:50:52.580542 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:52.580561 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:52.580658 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:52.581050 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:52.581054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:52.581075 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:52.581214 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:52.581224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.582330 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.582333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:52.583258 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:52.583267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.584142 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.584146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:52.584268 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:52.584298 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 14:50:52.584419 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 14:50:52.584504 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 14:50:52.584728 23852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.580496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:60847 (local address 127.8.221.195:45859)
0620 14:50:52.580647 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:52.580652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:52.580689 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:50:52.580761 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:52.580766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:52.580825 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:52.580944 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:52.581378 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.582191 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.583391 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.584011 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.584109 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:52.584161 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:52.584433 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 14:50:52.584496 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:50:52.585232 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.580723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:40425 (local address 127.8.221.194:58207)
0620 14:50:52.581135 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:52.581155 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:52.581247 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:52.581582 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:52.581585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:52.581624 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:52.581760 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:52.581767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.582724 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.582727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:52.583686 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:52.583696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.584619 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.584623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:52.584882 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:52.584917 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:50:52.585008 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:50:52.585085 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":135,"threads_started":1}
I20260620 14:50:52.585292 23863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.580900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:58207 (local address 127.8.221.193:40425)
0620 14:50:52.581167 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:52.581173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:52.581207 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:50:52.581379 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:52.581383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:52.581449 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:52.581544 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:52.581927 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.582592 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.583827 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.584481 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.584611 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:52.584760 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:52.585030 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 14:50:52.585146 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":71,"threads_started":1}
I20260620 14:50:52.585489 23796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c108d3047ba4f7c9260c480ee693dae" candidate_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1e55097d214ce4bc7cceebf840fc36" is_pre_election: true
I20260620 14:50:52.585886 23796 raft_consensus.cc:2454] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9301e8263f34d22a8ad809ee4a2ed48 in term 0.
I20260620 14:50:52.585979 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c108d3047ba4f7c9260c480ee693dae" candidate_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" is_pre_election: true
I20260620 14:50:52.586396 23526 raft_consensus.cc:2454] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9301e8263f34d22a8ad809ee4a2ed48 in term 0.
I20260620 14:50:52.586444 23597 leader_election.cc:302] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9301e8263f34d22a8ad809ee4a2ed48, db1e55097d214ce4bc7cceebf840fc36; no voters:
I20260620 14:50:52.586783 23861 raft_consensus.cc:2781] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:50:52.586874 23861 raft_consensus.cc:482] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:50:52.586905 23861 raft_consensus.cc:3037] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:52.588160 23861 raft_consensus.cc:504] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } }
I20260620 14:50:52.588608 23861 leader_election.cc:288] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [CANDIDATE]: Term 1 election: Requested vote from peers db1e55097d214ce4bc7cceebf840fc36 (127.8.221.195:45859), aff3fa9f4ed64017aad4d7f2804c0869 (127.8.221.193:40425)
I20260620 14:50:52.588948 23796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c108d3047ba4f7c9260c480ee693dae" candidate_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db1e55097d214ce4bc7cceebf840fc36"
I20260620 14:50:52.589028 23526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c108d3047ba4f7c9260c480ee693dae" candidate_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff3fa9f4ed64017aad4d7f2804c0869"
I20260620 14:50:52.589095 23796 raft_consensus.cc:3037] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:52.589208 23526 raft_consensus.cc:3037] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:52.590536 23796 raft_consensus.cc:2454] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9301e8263f34d22a8ad809ee4a2ed48 in term 1.
I20260620 14:50:52.590749 23526 raft_consensus.cc:2454] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9301e8263f34d22a8ad809ee4a2ed48 in term 1.
I20260620 14:50:52.590867 23597 leader_election.cc:302] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9301e8263f34d22a8ad809ee4a2ed48, db1e55097d214ce4bc7cceebf840fc36; no voters:
I20260620 14:50:52.591070 23861 raft_consensus.cc:2781] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:52.591296 23861 raft_consensus.cc:686] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [term 1 LEADER]: Becoming Leader. State: Replica: a9301e8263f34d22a8ad809ee4a2ed48, State: Running, Role: LEADER
I20260620 14:50:52.591630 23861 consensus_queue.cc:237] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } }
I20260620 14:50:52.594691 23383 catalog_manager.cc:5495] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9301e8263f34d22a8ad809ee4a2ed48 (127.8.221.194). New cstate: current_term: 1 leader_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } health_report { overall_health: UNKNOWN } } }
I20260620 14:50:52.616950 23853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.612504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35182 (local address 127.8.221.194:36261)
0620 14:50:52.612662 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:52.612668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:52.612685 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:50:52.612808 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:52.612811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:52.612879 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:52.612992 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:52.613262 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.613918 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.614797 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:52.615109 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:52.615285 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:52.615628 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:52.616358 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:52.616505 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:52.616626 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 14:50:52.616741 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:50:52.624250 23526 raft_consensus.cc:1261] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Refusing update from remote peer a9301e8263f34d22a8ad809ee4a2ed48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:52.624408 23796 raft_consensus.cc:1261] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36 [term 1 FOLLOWER]: Refusing update from remote peer a9301e8263f34d22a8ad809ee4a2ed48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:52.625204 23861 consensus_queue.cc:1035] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:52.625550 23864 consensus_queue.cc:1035] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:52.636267 23870 mvcc.cc:205] Tried to move back new op lower bound from 7298937047540609024 to 7298937047418163200. Current Snapshot: MvccSnapshot[applied={T|T < 7298937047540609024 or (T in {7298937047540609024})}]
I20260620 14:50:52.642405 23383 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:52.642642 23383 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:52.645507 23383 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb: Sending DeleteTablet for 3 replicas of tablet 3c108d3047ba4f7c9260c480ee693dae
I20260620 14:50:52.646593 23644 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c108d3047ba4f7c9260c480ee693dae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:52 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:35174
I20260620 14:50:52.646596 23776 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c108d3047ba4f7c9260c480ee693dae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:52 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:51490
I20260620 14:50:52.646696 23506 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c108d3047ba4f7c9260c480ee693dae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:52 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:51724
I20260620 14:50:52.647248 23876 tablet_replica.cc:330] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869: stopping tablet replica
I20260620 14:50:52.647243 23877 tablet_replica.cc:330] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36: stopping tablet replica
I20260620 14:50:52.647377 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 23422
I20260620 14:50:52.647403 23876 raft_consensus.cc:2229] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:52.648193 23877 raft_consensus.cc:2229] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:52.648531 23878 tablet_replica.cc:330] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48: stopping tablet replica
I20260620 14:50:52.648582 23877 raft_consensus.cc:2258] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:52.648802 23878 raft_consensus.cc:2229] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:50:52.649219 23878 raft_consensus.cc:2258] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:52.650743 23877 ts_tablet_manager.cc:1905] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:52.651693 23878 ts_tablet_manager.cc:1905] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:52.654211 23877 ts_tablet_manager.cc:1918] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:52.654340 23877 log.cc:1196] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/3c108d3047ba4f7c9260c480ee693dae
I20260620 14:50:52.654773 23877 ts_tablet_manager.cc:1939] T 3c108d3047ba4f7c9260c480ee693dae P db1e55097d214ce4bc7cceebf840fc36: Deleting consensus metadata
W20260620 14:50:52.654844 23368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.221.193:40425 (error 108)
I20260620 14:50:52.655248 23878 ts_tablet_manager.cc:1918] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:52.655359 23878 log.cc:1196] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/3c108d3047ba4f7c9260c480ee693dae
I20260620 14:50:52.655745 23878 ts_tablet_manager.cc:1939] T 3c108d3047ba4f7c9260c480ee693dae P a9301e8263f34d22a8ad809ee4a2ed48: Deleting consensus metadata
I20260620 14:50:52.656055 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 23577
I20260620 14:50:52.657569 23368 catalog_manager.cc:4841] TS db1e55097d214ce4bc7cceebf840fc36 (127.8.221.195:45859): tablet 3c108d3047ba4f7c9260c480ee693dae (table test-table [id=d7282edd79d7407ab8351851d5ab1cae]) successfully deleted
I20260620 14:50:52.658668 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.658361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:40425 (local address 127.0.0.1:51734)
0620 14:50:52.658487 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:52.658576 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.193:40425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 14:50:52.658842 23368 catalog_manager.cc:4584] TS aff3fa9f4ed64017aad4d7f2804c0869 (127.8.221.193:40425): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c108d3047ba4f7c9260c480ee693dae: Network error: Client connection negotiation failed: client connection to 127.8.221.193:40425: connect: Connection refused (error 111)
I20260620 14:50:52.661738 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 23713
I20260620 14:50:52.664112 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.663766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:36261 (local address 127.0.0.1:35198)
0620 14:50:52.663955 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:52.664029 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.194:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 14:50:52.668692 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 23351
I20260620 14:50:52.676779 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:43407
--webserver_interface=127.8.221.254
--webserver_port=41951
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:43407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:52.815358 23880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:52.815789 23880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:52.815886 23880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:52.820462 23880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:52.820552 23880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:52.820573 23880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:52.820593 23880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:52.820611 23880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:52.827045 23880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:43407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:43407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=41951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:52.828444 23880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:52.829669 23880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:52.839691 23885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:52.839771 23888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:52.839787 23886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:52.840186 23880 server_base.cc:1034] running on GCE node
I20260620 14:50:52.840909 23880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:52.842538 23880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:52.843780 23880 hybrid_clock.cc:648] HybridClock initialized: now 1781967052843744 us; error 54 us; skew 500 ppm
Jun 20 14:50:52 dist-test-slave-65tz krb5kdc[22804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967052, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:52.847275 23880 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:50:52.848459 23880 webserver.cc:460] Webserver started at http://127.8.221.254:41951/ using document root <none> and password file <none>
I20260620 14:50:52.849167 23880 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:52.849284 23880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:52.854331 23880 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:52.857322 23895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:52.859004 23880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:52.859169 23880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "1621fd4489704ef9a2f1f48d87190fbb"
format_stamp: "Formatted at 2026-06-20 14:50:50 on dist-test-slave-65tz"
server_key: "0deb52d1f9e8b26ae72a632ba2444dad"
server_key_iv: "b08d1301e48649c1c4f5bd3d575207f1"
server_key_version: "encryptionkey@0"
I20260620 14:50:52.859900 23880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:52.883278 23880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:52.887161 23880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:52.887439 23880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:52.896934 23880 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:43407
I20260620 14:50:52.897284 23947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:43407 every 8 connection(s)
I20260620 14:50:52.898504 23880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:52.903254 23948 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:50:52.905057 23948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb: Bootstrap starting.
I20260620 14:50:52.905076 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 23880
I20260620 14:50:52.905586 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:40425
--local_ip_for_outbound_sockets=127.8.221.193
--tserver_master_addrs=127.8.221.254:43407
--webserver_port=34955
--webserver_interface=127.8.221.193
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:50:52.923280 23948 log.cc:824] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:52.934438 23948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:50:52.934861 23948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb: Bootstrap complete.
I20260620 14:50:52.939029 23948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1621fd4489704ef9a2f1f48d87190fbb" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43407 } }
I20260620 14:50:52.939430 23948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1621fd4489704ef9a2f1f48d87190fbb, State: Initialized, Role: FOLLOWER
I20260620 14:50:52.939930 23948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1621fd4489704ef9a2f1f48d87190fbb" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43407 } }
I20260620 14:50:52.940047 23948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:52.940140 23948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:52.940263 23948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:50:52.942497 23948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1621fd4489704ef9a2f1f48d87190fbb" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43407 } }
I20260620 14:50:52.942860 23948 leader_election.cc:302] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1621fd4489704ef9a2f1f48d87190fbb; no voters:
I20260620 14:50:52.943248 23948 leader_election.cc:288] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:50:52.943329 23952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:50:52.943534 23952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [term 3 LEADER]: Becoming Leader. State: Replica: 1621fd4489704ef9a2f1f48d87190fbb, State: Running, Role: LEADER
I20260620 14:50:52.943845 23952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1621fd4489704ef9a2f1f48d87190fbb" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43407 } }
I20260620 14:50:52.944506 23948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:52.944777 23952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1621fd4489704ef9a2f1f48d87190fbb. Latest consensus state: current_term: 3 leader_uuid: "1621fd4489704ef9a2f1f48d87190fbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1621fd4489704ef9a2f1f48d87190fbb" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43407 } } }
I20260620 14:50:52.944898 23952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:52.945283 23956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:52.945585 23953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1621fd4489704ef9a2f1f48d87190fbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1621fd4489704ef9a2f1f48d87190fbb" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43407 } } }
I20260620 14:50:52.945901 23953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1621fd4489704ef9a2f1f48d87190fbb [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:52.950663 23956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:52.951404 23956 catalog_manager.cc:1259] Loaded cluster ID: 43eae1cb22eb4f96a1618abe483e3adf
I20260620 14:50:52.951465 23956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 14:50:52.952608 23956 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff58bdbb9ad google::LogMessage::Fail() at ??:0
@ 0x7ff58bdbe88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff58bdbb4de google::LogMessage::Flush() at ??:0
@ 0x7ff58bdbca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff5966fafdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7ff5966fbd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff5966f8579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7ff59673c640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff58e00d3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff58e00dc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7ff58e00f3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff58dffe512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff58f6426db start_thread at ??:0
@ 0x7ff58a8b961f clone at ??:0
W20260620 14:50:53.077214 23950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:53.077553 23950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:53.077603 23950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:53.083426 23950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:53.083529 23950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:53.083627 23950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:50:53.090479 23950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:40425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=34955
--tserver_master_addrs=127.8.221.254:43407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:53.091975 23950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:53.093336 23950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:53.105135 23989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:53.105381 23990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:53.105131 23992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:53.106177 23950 server_base.cc:1034] running on GCE node
I20260620 14:50:53.106695 23950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:53.107570 23950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:53.108767 23950 hybrid_clock.cc:648] HybridClock initialized: now 1781967053108739 us; error 45 us; skew 500 ppm
Jun 20 14:50:53 dist-test-slave-65tz krb5kdc[22804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967053, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:53.112026 23950 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:50:53.113217 23950 webserver.cc:460] Webserver started at http://127.8.221.193:34955/ using document root <none> and password file <none>
I20260620 14:50:53.114059 23950 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:53.114161 23950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:53.119141 23950 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:50:53.122210 23999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:53.123920 23950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:53.124082 23950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "aff3fa9f4ed64017aad4d7f2804c0869"
format_stamp: "Formatted at 2026-06-20 14:50:50 on dist-test-slave-65tz"
server_key: "f1083f14ef0729e5b606d9c8d91f9c95"
server_key_iv: "7db513b06b2198e38e54e8c4f08c3007"
server_key_version: "encryptionkey@0"
I20260620 14:50:53.124743 23950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:53.148378 23950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:53.152706 23950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:53.152961 23950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:53.153681 23950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:53.155125 24006 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:50:53.162243 23950 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:50:53.162382 23950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20260620 14:50:53.162484 23950 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:50:53.164573 23950 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:50:53.164645 23950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 14:50:53.164917 24006 tablet_bootstrap.cc:492] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869: Bootstrap starting.
I20260620 14:50:53.176127 23950 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:40425
I20260620 14:50:53.176537 24113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:40425 every 8 connection(s)
I20260620 14:50:53.177498 23950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:50:53.185186 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 23950
I20260620 14:50:53.185767 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:36261
--local_ip_for_outbound_sockets=127.8.221.194
--tserver_master_addrs=127.8.221.254:43407
--webserver_port=37053
--webserver_interface=127.8.221.194
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:50:53.186291 24006 log.cc:824] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:53.195600 24006 tablet_bootstrap.cc:492] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:50:53.196077 24006 tablet_bootstrap.cc:492] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869: Bootstrap complete.
I20260620 14:50:53.197580 24006 ts_tablet_manager.cc:1397] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.008s
I20260620 14:50:53.201546 24006 raft_consensus.cc:348] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } }
I20260620 14:50:53.202131 24006 raft_consensus.cc:729] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff3fa9f4ed64017aad4d7f2804c0869, State: Initialized, Role: FOLLOWER
I20260620 14:50:53.202677 24006 consensus_queue.cc:260] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } }
I20260620 14:50:53.203711 24006 ts_tablet_manager.cc:1428] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 14:50:53.204936 24116 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:50:53.180470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:46129)
0620 14:50:53.181266 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:53.181343 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:53.182561 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:53.203829 (+ 21268us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":213,"threads_started":1}
W20260620 14:50:53.205660 24114 heartbeater.cc:644] Failed to heartbeat to 127.8.221.254:43407 (0 consecutive failures): Network error: Failed to ping master at 127.8.221.254:43407: Client connection negotiation failed: client connection to 127.8.221.254:43407: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:50:53.206529 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:53.206242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:38741)
0620 14:50:53.206383 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:53.206465 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 14:50:53.207500 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:53.207293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:33121)
0620 14:50:53.207413 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:53.207455 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 14:50:53.207757 24114 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:50:53.308334 24118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:53.308600 24118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:53.308643 24118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:53.312722 24118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:53.312800 24118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:53.312860 24118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:50:53.318490 24118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:36261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=37053
--tserver_master_addrs=127.8.221.254:43407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:53.319607 24118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:53.320674 24118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:53.331157 24127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:53.331171 24124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:53.331359 24118 server_base.cc:1034] running on GCE node
W20260620 14:50:53.331558 24125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:53.332032 24118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:53.332747 24118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:53.333884 24118 hybrid_clock.cc:648] HybridClock initialized: now 1781967053333860 us; error 54 us; skew 500 ppm
Jun 20 14:50:53 dist-test-slave-65tz krb5kdc[22804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967053, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:53.336995 24118 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:50:53.338088 24118 webserver.cc:460] Webserver started at http://127.8.221.194:37053/ using document root <none> and password file <none>
I20260620 14:50:53.338693 24118 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:53.338784 24118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:53.343132 24118 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:50:53.345568 24134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:53.347107 24118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:50:53.347262 24118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "a9301e8263f34d22a8ad809ee4a2ed48"
format_stamp: "Formatted at 2026-06-20 14:50:50 on dist-test-slave-65tz"
server_key: "73f1ad3dfd21fbb43b695194e883c73c"
server_key_iv: "b46e0fbe8d9992d355a0bb3db693579e"
server_key_version: "encryptionkey@0"
I20260620 14:50:53.347833 24118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:53.360694 24118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:53.363659 24118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:53.363862 24118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:53.364494 24118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:53.365307 24118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:53.365358 24118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:53.365425 24118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:53.365473 24118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:53.376165 24118 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:36261
I20260620 14:50:53.376271 24247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:36261 every 8 connection(s)
I20260620 14:50:53.377218 24118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:50:53.382122 24250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:53.379935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:40459)
0620 14:50:53.380491 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:53.380643 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":138,"threads_started":1}
W20260620 14:50:53.382746 24248 heartbeater.cc:644] Failed to heartbeat to 127.8.221.254:43407 (0 consecutive failures): Network error: Failed to ping master at 127.8.221.254:43407: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
I20260620 14:50:53.383507 24250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:53.383261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:55467)
0620 14:50:53.383383 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:53.383441 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 14:50:53.384390 24250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:53.384107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:35355)
0620 14:50:53.384224 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:53.384317 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 14:50:53.384685 24248 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:50:53.384719 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 24118
I20260620 14:50:53.385191 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:45859
--local_ip_for_outbound_sockets=127.8.221.195
--tserver_master_addrs=127.8.221.254:43407
--webserver_port=33143
--webserver_interface=127.8.221.195
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:53.511475 24251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:53.511795 24251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:53.511897 24251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:53.516176 24251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:53.516258 24251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:53.516327 24251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:50:53.522317 24251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:45859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=33143
--tserver_master_addrs=127.8.221.254:43407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:50:53.523685 24251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:53.524894 24251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:53.536520 24256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:53.536520 24257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:53.536615 24259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:53.536881 24251 server_base.cc:1034] running on GCE node
I20260620 14:50:53.537395 24251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:53.538239 24251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:53.539438 24251 hybrid_clock.cc:648] HybridClock initialized: now 1781967053539418 us; error 36 us; skew 500 ppm
Jun 20 14:50:53 dist-test-slave-65tz krb5kdc[22804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967053, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:53.542791 24251 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:50:53.543926 24251 webserver.cc:460] Webserver started at http://127.8.221.195:33143/ using document root <none> and password file <none>
I20260620 14:50:53.544625 24251 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:53.544723 24251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:53.549518 24251 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:50:53.552421 24266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:53.554113 24251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:53.554266 24251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "db1e55097d214ce4bc7cceebf840fc36"
format_stamp: "Formatted at 2026-06-20 14:50:51 on dist-test-slave-65tz"
server_key: "1813c2bd82f8e70941934bd36a4c3904"
server_key_iv: "a5b39f1a2b663927aa166b46b7355302"
server_key_version: "encryptionkey@0"
I20260620 14:50:53.554963 24251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:53.574694 24251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:53.578465 24251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:53.578734 24251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:53.579450 24251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:53.580427 24251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:53.580502 24251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:53.580582 24251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:53.580633 24251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:53.593375 24251 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:45859
I20260620 14:50:53.593461 24379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:45859 every 8 connection(s)
I20260620 14:50:53.594766 24251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:50:53.599459 24382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:53.597209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.195:34267)
0620 14:50:53.597899 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:53.598054 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":175,"threads_started":1}
W20260620 14:50:53.600121 24380 heartbeater.cc:644] Failed to heartbeat to 127.8.221.254:43407 (0 consecutive failures): Network error: Failed to ping master at 127.8.221.254:43407: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
I20260620 14:50:53.600878 24382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:53.600619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.195:42029)
0620 14:50:53.600753 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:53.600809 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 14:50:53.601621 24382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:53.601429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.195:51233)
0620 14:50:53.601534 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:53.601576 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 14:50:53.601872 24380 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:50:53.603737 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 24251
I20260620 14:50:54.209008 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:54.208408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:41841)
0620 14:50:54.208758 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:54.208857 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":82,"threads_started":1}
I20260620 14:50:54.386009 24385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:54.385447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:51507)
0620 14:50:54.385829 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:54.385903 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":109,"threads_started":1}
I20260620 14:50:54.603092 24386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:54.602594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.195:42801)
0620 14:50:54.602901 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:54.602981 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":96,"threads_started":1}
I20260620 14:50:54.816054 24387 raft_consensus.cc:482] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:50:54.816288 24387 raft_consensus.cc:504] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } }
I20260620 14:50:54.817778 24387 leader_election.cc:288] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1e55097d214ce4bc7cceebf840fc36 (127.8.221.195:45859), a9301e8263f34d22a8ad809ee4a2ed48 (127.8.221.194:36261)
Jun 20 14:50:54 dist-test-slave-65tz krb5kdc[22804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967053, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.194@KRBTEST.COM
Jun 20 14:50:54 dist-test-slave-65tz krb5kdc[22804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967053, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.195@KRBTEST.COM
I20260620 14:50:54.833981 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:54.818621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:48323 (local address 127.8.221.194:36261)
0620 14:50:54.818979 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:54.819004 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:54.819057 (+ 53us) server_negotiation.cc:407] Connection header received
0620 14:50:54.819982 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:54.819992 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:54.820419 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:54.820750 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:54.822254 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:54.823280 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:54.824666 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:54.825154 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:54.825341 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:54.828426 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:54.828433 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:54.828446 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:54.828487 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:54.830995 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:54.831676 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:54.831685 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:54.831693 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:54.831851 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:54.832296 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:54.832301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:54.832303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:54.832777 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:54.832945 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:54.833464 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 14:50:54.833758 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":104,"threads_started":1}
I20260620 14:50:54.833978 24389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:54.818196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:45859 (local address 127.8.221.193:34289)
0620 14:50:54.818574 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:54.818594 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:54.818732 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:54.820781 (+ 2049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:54.820789 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:54.821434 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:54.822073 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:54.822092 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:54.823426 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:54.823429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:54.824498 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:54.824508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:54.825250 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:54.825254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:54.825506 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:54.826350 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:54.826363 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:54.828552 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:54.831415 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:54.831418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:54.831420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:54.831664 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:54.832109 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:54.832112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:54.832113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:54.832192 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:54.832861 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:54.832870 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:54.833125 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 14:50:54.833340 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 14:50:54.833714 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":209,"spinlock_wait_cycles":21888,"thread_start_us":82,"threads_started":1}
I20260620 14:50:54.833981 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:54.818353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:34289 (local address 127.8.221.195:45859)
0620 14:50:54.818839 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:54.818869 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:54.818924 (+ 55us) server_negotiation.cc:407] Connection header received
0620 14:50:54.819764 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:54.819770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:54.820223 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:54.820598 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:54.822245 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:54.823270 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:54.824622 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:54.825120 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:54.825302 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:54.828734 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:54.828743 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:54.828758 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:54.828792 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:54.830993 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:54.831786 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:54.831791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:54.831830 (+ 39us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:54.831997 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:54.832298 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:54.832301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:54.832306 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:54.832732 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:54.832852 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:54.833447 (+ 595us) server_negotiation.cc:299] Negotiation successful
0620 14:50:54.833749 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":168,"threads_started":1}
I20260620 14:50:54.834355 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:54.818067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:36261 (local address 127.8.221.193:48323)
0620 14:50:54.818938 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:54.818958 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:54.819085 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:54.820779 (+ 1694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:54.820787 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:54.821432 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:54.822071 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:54.822091 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:54.823436 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:54.823440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:54.824504 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:54.824513 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:54.825288 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:54.825291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:54.825497 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:54.826288 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:54.826306 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:54.828101 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:54.831135 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:54.831141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:54.831157 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:54.831509 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:54.831973 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:54.831977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:54.831979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:54.832182 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:54.832932 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:54.832935 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:54.833126 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 14:50:54.833338 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 14:50:54.833848 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":334,"spinlock_wait_cycles":17152,"thread_start_us":87,"threads_started":1}
W20260620 14:50:54.837067 24003 leader_election.cc:341] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9301e8263f34d22a8ad809ee4a2ed48 (127.8.221.194:36261): Not found: Tablet not found: 3c108d3047ba4f7c9260c480ee693dae
W20260620 14:50:54.837307 24000 leader_election.cc:341] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db1e55097d214ce4bc7cceebf840fc36 (127.8.221.195:45859): Not found: Tablet not found: 3c108d3047ba4f7c9260c480ee693dae
I20260620 14:50:54.837430 24000 leader_election.cc:302] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aff3fa9f4ed64017aad4d7f2804c0869; no voters: a9301e8263f34d22a8ad809ee4a2ed48, db1e55097d214ce4bc7cceebf840fc36
I20260620 14:50:54.837747 24387 raft_consensus.cc:2726] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:50:55.210443 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:55.209985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:54385)
0620 14:50:55.210206 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:55.210315 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 14:50:55.387523 24392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:55.386925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:39447)
0620 14:50:55.387253 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:55.387332 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":90,"threads_started":1}
I20260620 14:50:55.604733 24393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:55.604093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.195:45641)
0620 14:50:55.604469 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:55.604578 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":97,"threads_started":1}
I20260620 14:50:56.211978 24394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:56.211435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:37689)
0620 14:50:56.211767 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:56.211846 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":77,"threads_started":1}
I20260620 14:50:56.389096 24395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:56.388567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:37431)
0620 14:50:56.388903 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:56.388967 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":118,"threads_started":1}
I20260620 14:50:56.606353 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:56.605819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.195:53139)
0620 14:50:56.606152 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:56.606226 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":98,"threads_started":1}
I20260620 14:50:56.718178 24397 raft_consensus.cc:482] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:50:56.718328 24397 raft_consensus.cc:504] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } }
I20260620 14:50:56.718765 24397 leader_election.cc:288] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1e55097d214ce4bc7cceebf840fc36 (127.8.221.195:45859), a9301e8263f34d22a8ad809ee4a2ed48 (127.8.221.194:36261)
W20260620 14:50:56.719758 24003 leader_election.cc:341] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9301e8263f34d22a8ad809ee4a2ed48 (127.8.221.194:36261): Not found: Tablet not found: 3c108d3047ba4f7c9260c480ee693dae
W20260620 14:50:56.719919 24000 leader_election.cc:341] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db1e55097d214ce4bc7cceebf840fc36 (127.8.221.195:45859): Not found: Tablet not found: 3c108d3047ba4f7c9260c480ee693dae
I20260620 14:50:56.719980 24003 leader_election.cc:302] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aff3fa9f4ed64017aad4d7f2804c0869; no voters: a9301e8263f34d22a8ad809ee4a2ed48, db1e55097d214ce4bc7cceebf840fc36
I20260620 14:50:56.720136 24397 raft_consensus.cc:2726] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:50:57.213621 24398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.212988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:41607)
0620 14:50:57.213356 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:57.213473 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":132,"threads_started":1}
I20260620 14:50:57.390776 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.390264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:34283)
0620 14:50:57.390593 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:57.390670 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
I20260620 14:50:57.607656 24400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.607173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.195:51335)
0620 14:50:57.607487 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:57.607561 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":103,"threads_started":1}
I20260620 14:50:58.215552 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:58.214931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:46855)
0620 14:50:58.215356 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:58.215440 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":103,"threads_started":1}
I20260620 14:50:58.392527 24402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:58.391927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:44325)
0620 14:50:58.392317 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:58.392400 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":92,"threads_started":1}
I20260620 14:50:58.609330 24403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:58.608781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.195:58985)
0620 14:50:58.609147 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:58.609229 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":129,"threads_started":1}
I20260620 14:50:58.862466 24404 raft_consensus.cc:482] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:50:58.862718 24404 raft_consensus.cc:504] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } }
I20260620 14:50:58.863327 24404 leader_election.cc:288] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1e55097d214ce4bc7cceebf840fc36 (127.8.221.195:45859), a9301e8263f34d22a8ad809ee4a2ed48 (127.8.221.194:36261)
W20260620 14:50:58.864265 24000 leader_election.cc:341] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db1e55097d214ce4bc7cceebf840fc36 (127.8.221.195:45859): Not found: Tablet not found: 3c108d3047ba4f7c9260c480ee693dae
W20260620 14:50:58.864387 24003 leader_election.cc:341] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9301e8263f34d22a8ad809ee4a2ed48 (127.8.221.194:36261): Not found: Tablet not found: 3c108d3047ba4f7c9260c480ee693dae
I20260620 14:50:58.864547 24003 leader_election.cc:302] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aff3fa9f4ed64017aad4d7f2804c0869; no voters: a9301e8263f34d22a8ad809ee4a2ed48, db1e55097d214ce4bc7cceebf840fc36
I20260620 14:50:58.864789 24404 raft_consensus.cc:2726] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:50:59.217041 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.216484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:58459)
0620 14:50:59.216825 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:59.216910 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":86,"threads_started":1}
I20260620 14:50:59.394343 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.393668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:40043)
0620 14:50:59.394044 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:59.394119 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":144,"threads_started":1}
I20260620 14:50:59.610773 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.610215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.195:52889)
0620 14:50:59.610557 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:59.610635 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":89,"threads_started":1}
I20260620 14:51:00.218711 24408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.218148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:55703)
0620 14:51:00.218485 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:00.218578 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":95,"threads_started":1}
I20260620 14:51:00.396209 24409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.395412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:39999)
0620 14:51:00.395922 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:00.396035 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":102,"threads_started":1}
I20260620 14:51:00.612409 24410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.611776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.195:48519)
0620 14:51:00.612197 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:00.612279 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":95,"threads_started":1}
I20260620 14:51:01.220525 24411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:01.219872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:41595)
0620 14:51:01.220277 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:01.220380 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":86,"threads_started":1}
I20260620 14:51:01.397845 24412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:01.397254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:40575)
0620 14:51:01.397575 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:01.397687 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":84,"threads_started":1}
I20260620 14:51:01.614233 24413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:01.613510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.195:35875)
0620 14:51:01.613991 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:01.614091 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":111,"threads_started":1}
I20260620 14:51:02.222391 24414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.221691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:58653)
0620 14:51:02.222164 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.222252 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":178,"threads_started":1}
I20260620 14:51:02.399521 24415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.398915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:34801)
0620 14:51:02.399289 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.399380 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":88,"threads_started":1}
I20260620 14:51:02.615887 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.615272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.195:53403)
0620 14:51:02.615661 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.615748 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":104,"threads_started":1}
I20260620 14:51:03.224134 24417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:03.223556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:55269)
0620 14:51:03.223916 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:03.224001 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
I20260620 14:51:03.382223 24418 raft_consensus.cc:482] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:03.382378 24418 raft_consensus.cc:504] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } }
I20260620 14:51:03.382920 24418 leader_election.cc:288] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1e55097d214ce4bc7cceebf840fc36 (127.8.221.195:45859), a9301e8263f34d22a8ad809ee4a2ed48 (127.8.221.194:36261)
W20260620 14:51:03.384097 24003 leader_election.cc:341] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9301e8263f34d22a8ad809ee4a2ed48 (127.8.221.194:36261): Not found: Tablet not found: 3c108d3047ba4f7c9260c480ee693dae
W20260620 14:51:03.384298 24000 leader_election.cc:341] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db1e55097d214ce4bc7cceebf840fc36 (127.8.221.195:45859): Not found: Tablet not found: 3c108d3047ba4f7c9260c480ee693dae
I20260620 14:51:03.384405 24000 leader_election.cc:302] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aff3fa9f4ed64017aad4d7f2804c0869; no voters: a9301e8263f34d22a8ad809ee4a2ed48, db1e55097d214ce4bc7cceebf840fc36
I20260620 14:51:03.384605 24418 raft_consensus.cc:2726] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:51:03.401206 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:03.400667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:35319)
0620 14:51:03.401005 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:03.401079 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":93,"threads_started":1}
I20260620 14:51:03.617544 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:03.616941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.195:45009)
0620 14:51:03.617346 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:03.617433 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":106,"threads_started":1}
I20260620 14:51:04.225862 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:04.225054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:33833)
0620 14:51:04.225549 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:04.225662 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":100,"threads_started":1}
I20260620 14:51:04.403021 24422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:04.402427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:53675)
0620 14:51:04.402783 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:04.402885 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":86,"threads_started":1}
I20260620 14:51:04.619426 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:04.618743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.195:40385)
0620 14:51:04.619196 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:04.619299 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":135,"threads_started":1}
I20260620 14:51:05.227658 24424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:05.226993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:39073)
0620 14:51:05.227393 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:05.227498 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":83,"threads_started":1}
I20260620 14:51:05.404553 24425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:05.404002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:58855)
0620 14:51:05.404355 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:05.404433 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":106,"threads_started":1}
I20260620 14:51:05.620831 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:05.620287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.195:58285)
0620 14:51:05.620616 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:05.620697 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":104,"threads_started":1}
I20260620 14:51:06.229115 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.228451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:42841)
0620 14:51:06.228837 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:06.228960 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":82,"threads_started":1}
I20260620 14:51:06.406126 24428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.405497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:45597)
0620 14:51:06.405929 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:06.406016 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":102,"threads_started":1}
I20260620 14:51:06.622404 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.621856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.195:36303)
0620 14:51:06.622200 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:06.622293 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":102,"threads_started":1}
I20260620 14:51:07.230581 24430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:07.229969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:44811)
0620 14:51:07.230346 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:07.230444 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":81,"threads_started":1}
I20260620 14:51:07.407735 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:07.407185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:48263)
0620 14:51:07.407542 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:07.407630 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":107,"threads_started":1}
I20260620 14:51:07.624075 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:07.623506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.195:50963)
0620 14:51:07.623900 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:07.623973 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":142,"threads_started":1}
I20260620 14:51:08.232265 24433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:08.231760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.193:51969)
0620 14:51:08.232079 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:08.232156 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":103,"threads_started":1}
I20260620 14:51:08.409947 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:08.409104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43407 (local address 127.8.221.194:47205)
0620 14:51:08.409676 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:08.409754 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:43407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":114,"threads_started":1}
I20260620 14:51:08.504731 24435 raft_consensus.cc:482] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:08.504906 24435 raft_consensus.cc:504] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db1e55097d214ce4bc7cceebf840fc36" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 45859 } } peers { permanent_uuid: "a9301e8263f34d22a8ad809ee4a2ed48" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 36261 } } peers { permanent_uuid: "aff3fa9f4ed64017aad4d7f2804c0869" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 40425 } }
I20260620 14:51:08.505446 24435 leader_election.cc:288] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db1e55097d214ce4bc7cceebf840fc36 (127.8.221.195:45859), a9301e8263f34d22a8ad809ee4a2ed48 (127.8.221.194:36261)
W20260620 14:51:08.506544 24003 leader_election.cc:341] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9301e8263f34d22a8ad809ee4a2ed48 (127.8.221.194:36261): Not found: Tablet not found: 3c108d3047ba4f7c9260c480ee693dae
W20260620 14:51:08.506693 24000 leader_election.cc:341] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db1e55097d214ce4bc7cceebf840fc36 (127.8.221.195:45859): Not found: Tablet not found: 3c108d3047ba4f7c9260c480ee693dae
I20260620 14:51:08.506806 24000 leader_election.cc:302] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aff3fa9f4ed64017aad4d7f2804c0869; no voters: a9301e8263f34d22a8ad809ee4a2ed48, db1e55097d214ce4bc7cceebf840fc36
I20260620 14:51:08.507035 24435 raft_consensus.cc:2726] T 3c108d3047ba4f7c9260c480ee693dae P aff3fa9f4ed64017aad4d7f2804c0869 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:51:08.611272 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 23950
I20260620 14:51:08.618304 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 24118
I20260620 14:51:08.624249 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 24251
2026-06-20T14:51:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19753 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:51:08 dist-test-slave-65tz krb5kdc[24438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:51:08 dist-test-slave-65tz krb5kdc[24438](info): set up 2 sockets
Jun 20 14:51:08 dist-test-slave-65tz krb5kdc[24438](info): commencing operation
krb5kdc: starting...
W20260620 14:51:09.742542 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:51:09 dist-test-slave-65tz krb5kdc[24438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:51:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:51:09.881349 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38213
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:38213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:10.012409 24451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:10.012862 24451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:10.012943 24451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:10.017644 24451 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:51:10.017791 24451 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:51:10.017850 24451 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:51:10.017877 24451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:10.017916 24451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:10.017954 24451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:10.018013 24451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:10.018056 24451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:10.024451 24451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:38213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:51:10.026016 24451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:10.027328 24451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:10.038198 24457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:10.038197 24456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:10.038276 24451 server_base.cc:1034] running on GCE node
W20260620 14:51:10.038223 24459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:10.039346 24451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:10.040815 24451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:10.042006 24451 hybrid_clock.cc:648] HybridClock initialized: now 1781967070041988 us; error 42 us; skew 500 ppm
Jun 20 14:51:10 dist-test-slave-65tz krb5kdc[24438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967070, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:10.045425 24451 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:51:10.046633 24451 webserver.cc:460] Webserver started at http://127.8.221.254:46171/ using document root <none> and password file <none>
I20260620 14:51:10.047359 24451 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:10.047428 24451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:10.047636 24451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:10.050263 24451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "cdc7bbdfb08641b794835c384921cd7c"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "4ef63526ad5903b7a719f414cc74974c"
server_key_iv: "5e5d090e9e7b4b56596b9d9cb044b262"
server_key_version: "encryptionkey@0"
I20260620 14:51:10.050880 24451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "cdc7bbdfb08641b794835c384921cd7c"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "4ef63526ad5903b7a719f414cc74974c"
server_key_iv: "5e5d090e9e7b4b56596b9d9cb044b262"
server_key_version: "encryptionkey@0"
I20260620 14:51:10.055503 24451 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 14:51:10.058501 24466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:10.060302 24451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:51:10.060527 24451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "cdc7bbdfb08641b794835c384921cd7c"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "4ef63526ad5903b7a719f414cc74974c"
server_key_iv: "5e5d090e9e7b4b56596b9d9cb044b262"
server_key_version: "encryptionkey@0"
I20260620 14:51:10.060676 24451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:10.076488 24451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:10.080451 24451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:10.080705 24451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:10.089888 24451 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:38213
I20260620 14:51:10.090267 24518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:38213 every 8 connection(s)
I20260620 14:51:10.091135 24451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:10.094365 24519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:10.099817 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 24451
I20260620 14:51:10.099984 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:51:10.100303 9079 external_mini_cluster.cc:1428] Setting key 64dc1f0c8773299d8d33de3ee65ebd66
I20260620 14:51:10.101171 24519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c: Bootstrap starting.
I20260620 14:51:10.104018 24519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:10.104990 24519 log.cc:824] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:10.107625 24519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c: No bootstrap required, opened a new log
Jun 20 14:51:10 dist-test-slave-65tz krb5kdc[24438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:51:10.111752 24519 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7bbdfb08641b794835c384921cd7c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38213 } }
I20260620 14:51:10.112048 24519 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:10.112139 24519 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc7bbdfb08641b794835c384921cd7c, State: Initialized, Role: FOLLOWER
I20260620 14:51:10.112617 24519 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7bbdfb08641b794835c384921cd7c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38213 } }
I20260620 14:51:10.112800 24519 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:10.112893 24519 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:10.113020 24519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:10.114251 24519 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7bbdfb08641b794835c384921cd7c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38213 } }
I20260620 14:51:10.114641 24519 leader_election.cc:302] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdc7bbdfb08641b794835c384921cd7c; no voters:
I20260620 14:51:10.115041 24519 leader_election.cc:288] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:10.115106 24524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:10.115298 24524 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 1 LEADER]: Becoming Leader. State: Replica: cdc7bbdfb08641b794835c384921cd7c, State: Running, Role: LEADER
I20260620 14:51:10.115710 24524 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7bbdfb08641b794835c384921cd7c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38213 } }
I20260620 14:51:10.116406 24519 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:10.117429 24525 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdc7bbdfb08641b794835c384921cd7c. Latest consensus state: current_term: 1 leader_uuid: "cdc7bbdfb08641b794835c384921cd7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7bbdfb08641b794835c384921cd7c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38213 } } }
I20260620 14:51:10.117578 24525 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:10.117921 24526 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdc7bbdfb08641b794835c384921cd7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7bbdfb08641b794835c384921cd7c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38213 } } }
I20260620 14:51:10.118216 24526 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:10.119613 24532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:10.119843 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:10.102280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58704 (local address 127.8.221.254:38213)
0620 14:51:10.102843 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:10.102864 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:10.102914 (+ 50us) server_negotiation.cc:407] Connection header received
0620 14:51:10.103816 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:10.103842 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:10.104362 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:10.104801 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:10.105943 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.107409 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.108495 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.109199 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.109388 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:10.112498 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:10.112507 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:10.112540 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:10.112582 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:10.115087 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:10.115780 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:10.115786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:10.115793 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:10.115973 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:10.116290 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:10.116295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:10.116297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:10.116976 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:10.118055 (+ 1079us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:10.118228 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 14:51:10.118602 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":146,"threads_started":1}
I20260620 14:51:10.122967 24532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:10.129959 24532 catalog_manager.cc:1347] Generated new cluster ID: 2ff1698fa3c04f44ad6210cbe10e7a60
I20260620 14:51:10.130064 24532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:10.142738 24532 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:51:10.144368 24532 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:10.161177 24532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c: Generated new TSK 0
I20260620 14:51:10.162418 24532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
I20260620 14:51:10.197546 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38213
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:10.321501 24545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:10.321866 24545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:10.321982 24545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:10.326642 24545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:10.326725 24545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:10.326789 24545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:51:10.332557 24545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:51:10.333838 24545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:10.334926 24545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:10.345826 24550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:10.345826 24551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:10.346285 24545 server_base.cc:1034] running on GCE node
W20260620 14:51:10.345856 24553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:10.346911 24545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:10.347635 24545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:10.348869 24545 hybrid_clock.cc:648] HybridClock initialized: now 1781967070348808 us; error 60 us; skew 500 ppm
Jun 20 14:51:10 dist-test-slave-65tz krb5kdc[24438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967070, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:10.352020 24545 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:51:10.352996 24545 webserver.cc:460] Webserver started at http://127.8.221.193:44527/ using document root <none> and password file <none>
I20260620 14:51:10.353637 24545 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:10.353736 24545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:10.354004 24545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:10.356109 24545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "b92be6e49c874c1298cade013b065f51"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "3439e63d12178a49e12b492223efeeef"
server_key_iv: "c625ce12042ccb186525bbc77542df8d"
server_key_version: "encryptionkey@0"
I20260620 14:51:10.356685 24545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "b92be6e49c874c1298cade013b065f51"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "3439e63d12178a49e12b492223efeeef"
server_key_iv: "c625ce12042ccb186525bbc77542df8d"
server_key_version: "encryptionkey@0"
I20260620 14:51:10.360767 24545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:51:10.363643 24560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:10.365732 24545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:51:10.365937 24545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "b92be6e49c874c1298cade013b065f51"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "3439e63d12178a49e12b492223efeeef"
server_key_iv: "c625ce12042ccb186525bbc77542df8d"
server_key_version: "encryptionkey@0"
I20260620 14:51:10.366075 24545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:10.383551 24545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:10.387775 24545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:10.387972 24545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:10.388643 24545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:10.389717 24545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:10.389811 24545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:10.389919 24545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:10.389974 24545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:10.402038 24545 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:45499
I20260620 14:51:10.402063 24673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:45499 every 8 connection(s)
I20260620 14:51:10.403196 24545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:51:10.406656 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 24545
I20260620 14:51:10.406817 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:51:10.407153 9079 external_mini_cluster.cc:1428] Setting key 1e13cc17383da063cb01630809c5c4c5
Jun 20 14:51:10 dist-test-slave-65tz krb5kdc[24438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967070, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.194@KRBTEST.COM" created.
I20260620 14:51:10.420313 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:10.405426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:60179 (local address 127.8.221.254:38213)
0620 14:51:10.405599 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:10.405603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:10.407265 (+ 1662us) server_negotiation.cc:407] Connection header received
0620 14:51:10.408430 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:10.408435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:10.408491 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:10.408597 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:10.410502 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.411134 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.412552 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.412809 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.412945 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:10.415938 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:10.415941 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:10.415944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:10.415971 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:10.417895 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:10.418485 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:10.418488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:10.418490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:10.418575 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:10.418990 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:10.418994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:10.418996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:10.419216 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:10.419336 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:10.419960 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 14:51:10.420111 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:51:10.421422 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:10.405952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:60179)
0620 14:51:10.407006 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:10.407080 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:10.408139 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:10.408921 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:10.408930 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:10.409487 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:10.410302 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:10.410326 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.411313 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.411317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:10.412400 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:10.412412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.413004 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.413008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:10.413215 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:10.414023 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:10.414042 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:10.415730 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:10.418021 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:10.418028 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:10.418049 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:10.418360 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:10.418688 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:10.418691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:10.418693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:10.418877 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:10.419336 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:10.419345 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:10.419677 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 14:51:10.419903 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 14:51:10.420339 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":143,"threads_started":1}
I20260620 14:51:10.423183 24674 heartbeater.cc:347] Connected to a master server at 127.8.221.254:38213
I20260620 14:51:10.423511 24674 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:10.424156 24674 heartbeater.cc:511] Master 127.8.221.254:38213 requested a full tablet report, sending...
I20260620 14:51:10.426415 24483 ts_manager.cc:194] Registered new tserver with Master: b92be6e49c874c1298cade013b065f51 (127.8.221.193:45499)
I20260620 14:51:10.428272 24483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:60179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
I20260620 14:51:10.437090 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38213
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:10.551303 24679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:10.551584 24679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:10.551637 24679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:10.555835 24679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:10.555907 24679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:10.555970 24679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:51:10.561409 24679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:51:10.562495 24679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:10.563555 24679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:10.574193 24685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:10.574183 24687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:10.574185 24684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:10.574622 24679 server_base.cc:1034] running on GCE node
I20260620 14:51:10.575085 24679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:10.575780 24679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:10.577003 24679 hybrid_clock.cc:648] HybridClock initialized: now 1781967070576988 us; error 42 us; skew 500 ppm
Jun 20 14:51:10 dist-test-slave-65tz krb5kdc[24438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967070, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:10.580232 24679 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:51:10.581219 24679 webserver.cc:460] Webserver started at http://127.8.221.194:37501/ using document root <none> and password file <none>
I20260620 14:51:10.581876 24679 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:10.581985 24679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:10.582203 24679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:10.584283 24679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "4012070e7f0048a5b20486063abc30db"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "8ca0b741c9b79cf9721b330927db9d87"
server_key_iv: "a0de6a1873627d1bd21f05e28ef66043"
server_key_version: "encryptionkey@0"
I20260620 14:51:10.584843 24679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "4012070e7f0048a5b20486063abc30db"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "8ca0b741c9b79cf9721b330927db9d87"
server_key_iv: "a0de6a1873627d1bd21f05e28ef66043"
server_key_version: "encryptionkey@0"
I20260620 14:51:10.588872 24679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 14:51:10.591646 24694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:10.593155 24679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:51:10.593313 24679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "4012070e7f0048a5b20486063abc30db"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "8ca0b741c9b79cf9721b330927db9d87"
server_key_iv: "a0de6a1873627d1bd21f05e28ef66043"
server_key_version: "encryptionkey@0"
I20260620 14:51:10.593437 24679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:10.616808 24679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:10.620632 24679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:10.620858 24679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:10.621490 24679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:10.622454 24679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:10.622545 24679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:10.622642 24679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:10.622699 24679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:10.633039 24679 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:39433
I20260620 14:51:10.633117 24807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:39433 every 8 connection(s)
I20260620 14:51:10.634191 24679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:51:10.634905 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 24679
I20260620 14:51:10.635051 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:51:10.635464 9079 external_mini_cluster.cc:1428] Setting key a68a9d6be39db6d3583119230df1b7ad
WARNING: no policy specified for kudu/127.8.221.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.195@KRBTEST.COM" created.
Jun 20 14:51:10 dist-test-slave-65tz krb5kdc[24438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967070, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:51:10.656687 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:10.636691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:47065 (local address 127.8.221.254:38213)
0620 14:51:10.636875 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:10.636880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:10.638235 (+ 1355us) server_negotiation.cc:407] Connection header received
0620 14:51:10.640502 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:10.640507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:10.640592 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:10.640722 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:10.642695 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.643323 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.645480 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.645983 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.646125 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:10.652615 (+ 6490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:10.652619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:10.652622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:10.652663 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:10.654408 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:10.655039 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:10.655042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:10.655043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:10.655123 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:10.655508 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:10.655512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:10.655514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:10.655711 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:10.655824 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:10.656380 (+ 556us) server_negotiation.cc:299] Negotiation successful
0620 14:51:10.656506 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:51:10.657577 24810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:10.637240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:47065)
0620 14:51:10.637997 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:10.638079 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:10.640198 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:10.640955 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:10.640963 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:10.641592 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:10.642494 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:10.642518 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.643510 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.643517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:10.645273 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:10.645290 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.646194 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.646200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:10.646448 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:10.647247 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:10.647276 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:10.652336 (+ 5060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:10.654559 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:10.654566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:10.654584 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:10.654893 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:10.655230 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:10.655234 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:10.655236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:10.655391 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:10.655831 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:10.655840 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:10.656127 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 14:51:10.656325 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 14:51:10.656706 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":148,"threads_started":1}
I20260620 14:51:10.659472 24808 heartbeater.cc:347] Connected to a master server at 127.8.221.254:38213
I20260620 14:51:10.659776 24808 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:10.660382 24808 heartbeater.cc:511] Master 127.8.221.254:38213 requested a full tablet report, sending...
I20260620 14:51:10.661741 24483 ts_manager.cc:194] Registered new tserver with Master: 4012070e7f0048a5b20486063abc30db (127.8.221.194:39433)
I20260620 14:51:10.662456 24483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:47065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
I20260620 14:51:10.670588 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38213
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:10.811031 24813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:10.811367 24813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:10.811414 24813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:10.815578 24813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:10.815654 24813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:10.815717 24813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:51:10.821600 24813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:38213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:51:10.823043 24813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:10.824354 24813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:10.836305 24821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:10.836458 24813 server_base.cc:1034] running on GCE node
W20260620 14:51:10.836295 24818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:10.836305 24819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:10.837296 24813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:10.838292 24813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:10.839504 24813 hybrid_clock.cc:648] HybridClock initialized: now 1781967070839448 us; error 77 us; skew 500 ppm
Jun 20 14:51:10 dist-test-slave-65tz krb5kdc[24438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967070, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:10.843532 24813 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:51:10.844961 24813 webserver.cc:460] Webserver started at http://127.8.221.195:44845/ using document root <none> and password file <none>
I20260620 14:51:10.845852 24813 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:10.845943 24813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:10.846230 24813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:10.849112 24813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "1bfab1be9f36478bbf52b29235163b22"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "fb59d68d07fa8c5e512c568485f77a9a"
server_key_iv: "a0eea83d54321860fa484a4eeb9c542b"
server_key_version: "encryptionkey@0"
I20260620 14:51:10.850054 24813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "1bfab1be9f36478bbf52b29235163b22"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "fb59d68d07fa8c5e512c568485f77a9a"
server_key_iv: "a0eea83d54321860fa484a4eeb9c542b"
server_key_version: "encryptionkey@0"
I20260620 14:51:10.855748 24813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:51:10.859392 24828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:10.861158 24813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:51:10.861415 24813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "1bfab1be9f36478bbf52b29235163b22"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "fb59d68d07fa8c5e512c568485f77a9a"
server_key_iv: "a0eea83d54321860fa484a4eeb9c542b"
server_key_version: "encryptionkey@0"
I20260620 14:51:10.861580 24813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:10.887042 24813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:10.891284 24813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:10.891566 24813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:10.892258 24813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:10.893261 24813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:10.893316 24813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:10.893375 24813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:10.893401 24813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:10.905931 24813 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:40049
I20260620 14:51:10.905973 24941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:40049 every 8 connection(s)
I20260620 14:51:10.907274 24813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:51:10.910104 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 24813
I20260620 14:51:10.910256 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:51:10.910591 9079 external_mini_cluster.cc:1428] Setting key d173fca72dd0a6747b067caeafdd50b0
Jun 20 14:51:10 dist-test-slave-65tz krb5kdc[24438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967070, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:51:10.924135 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:10.909891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:36859 (local address 127.8.221.254:38213)
0620 14:51:10.910128 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:10.910136 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:10.911671 (+ 1535us) server_negotiation.cc:407] Connection header received
0620 14:51:10.912828 (+ 1157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:10.912831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:10.912885 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:10.912961 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:10.914367 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.914911 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.916087 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.916394 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.916500 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:10.919867 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:10.919870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:10.919872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:10.919896 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:10.921410 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:10.922186 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:10.922190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:10.922191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:10.922275 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:10.922721 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:10.922725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:10.922726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:10.922919 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:10.923035 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:10.923678 (+ 643us) server_negotiation.cc:299] Negotiation successful
0620 14:51:10.923913 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 14:51:10.925271 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:10.910476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.195:36859)
0620 14:51:10.911491 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:10.911571 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:10.912611 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:10.913157 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:10.913174 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:10.913634 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:10.914219 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:10.914233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.915051 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.915054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:10.915947 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:10.915956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.916566 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.916570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:10.916784 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:10.917660 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:10.917678 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:10.919658 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:10.921557 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:10.921563 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:10.921580 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:10.922049 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:10.922404 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:10.922408 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:10.922411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:10.922601 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:10.923042 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:10.923051 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:10.923324 (+ 273us) client_negotiation.cc:769] Sending connection context
0620 14:51:10.923530 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 14:51:10.923902 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":198,"threads_started":1}
I20260620 14:51:10.926671 24942 heartbeater.cc:347] Connected to a master server at 127.8.221.254:38213
I20260620 14:51:10.927049 24942 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:10.927785 24942 heartbeater.cc:511] Master 127.8.221.254:38213 requested a full tablet report, sending...
I20260620 14:51:10.929318 24483 ts_manager.cc:194] Registered new tserver with Master: 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049)
I20260620 14:51:10.930167 24483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:36859
I20260620 14:51:10.938957 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:51:10.950708 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:10.942198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58718 (local address 127.8.221.254:38213)
0620 14:51:10.942361 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:10.942365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:10.942507 (+ 142us) server_negotiation.cc:407] Connection header received
0620 14:51:10.942675 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:10.942677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:10.942737 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:10.942839 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:10.943811 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.944375 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.945356 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.945585 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.945681 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:10.947361 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:10.947363 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:10.947365 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:10.947389 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:10.948868 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:10.949355 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:10.949358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:10.949359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:10.949436 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:10.949722 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:10.949725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:10.949726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:10.949974 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:10.950095 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:10.950440 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 14:51:10.950568 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:51:10.954442 24483 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:51:10.957551 24483 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:10.976336 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:10.969661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55108 (local address 127.8.221.194:39433)
0620 14:51:10.970483 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:10.970496 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:10.970552 (+ 56us) server_negotiation.cc:407] Connection header received
0620 14:51:10.970639 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:10.970647 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:10.970967 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:10.971252 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:10.971525 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.972570 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.974650 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.975522 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.975831 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:10.976013 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:10.976067 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 14:51:10.976163 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":137,"threads_started":1}
I20260620 14:51:10.976351 24953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:10.968762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:39433 (local address 127.0.0.1:55108)
0620 14:51:10.969971 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:10.970026 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:10.970173 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:10.971142 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:10.971145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:10.971166 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:10.971389 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:10.971398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.972765 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.972771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:10.974502 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:10.974514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.975653 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.975657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:10.975840 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:10.975918 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 14:51:10.975991 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:51:10.976173 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"mutex_wait_us":308,"thread_start_us":119,"threads_started":1}
I20260620 14:51:10.976370 24954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:10.968762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:45499 (local address 127.0.0.1:58008)
0620 14:51:10.969966 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:10.970013 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:10.970175 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:10.970988 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:10.970995 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:10.971017 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:10.971476 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:10.971487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.973048 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.973052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:10.974510 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:10.974515 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.975577 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.975580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:10.975716 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:10.975795 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 14:51:10.975902 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:51:10.976128 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":364,"thread_start_us":76,"threads_started":1}
I20260620 14:51:10.977093 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:10.970083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46254 (local address 127.8.221.195:40049)
0620 14:51:10.970562 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:10.970574 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:10.970831 (+ 257us) server_negotiation.cc:407] Connection header received
0620 14:51:10.970958 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:10.970963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:10.971182 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:10.971331 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:10.972045 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.973144 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.974466 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.975420 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.975546 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:10.975732 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:10.976240 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 14:51:10.976361 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":205,"threads_started":1}
I20260620 14:51:10.977105 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:10.968964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58008 (local address 127.8.221.193:45499)
0620 14:51:10.969817 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:10.969827 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:10.970136 (+ 309us) server_negotiation.cc:407] Connection header received
0620 14:51:10.970490 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:10.970497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:10.970803 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:10.971013 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:10.971575 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.972915 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.974646 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.975420 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.976628 (+ 1208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:10.976807 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:10.976857 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 14:51:10.976964 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":144,"threads_started":1}
I20260620 14:51:10.977658 24956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:10.969475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:40049 (local address 127.0.0.1:46254)
0620 14:51:10.970721 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:10.970736 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:10.970837 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:10.971774 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:10.971776 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:10.971795 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:10.971891 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:10.971898 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.973282 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.973284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:10.974327 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:10.974337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:10.975794 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:10.975797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:10.975936 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:10.975957 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:51:10.976017 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:51:10.976128 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1177,"mutex_wait_us":39,"thread_start_us":136,"threads_started":1}
I20260620 14:51:10.980134 24742 tablet_service.cc:1467] Processing CreateTablet for tablet 4215c197525542bc8e386246a8c3375f (DEFAULT_TABLE table=test-table [id=c096fecd7f7047eab5362216eba325ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:10.980566 24876 tablet_service.cc:1467] Processing CreateTablet for tablet 4215c197525542bc8e386246a8c3375f (DEFAULT_TABLE table=test-table [id=c096fecd7f7047eab5362216eba325ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:10.980410 24608 tablet_service.cc:1467] Processing CreateTablet for tablet 4215c197525542bc8e386246a8c3375f (DEFAULT_TABLE table=test-table [id=c096fecd7f7047eab5362216eba325ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:10.981357 24742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4215c197525542bc8e386246a8c3375f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:10.981622 24608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4215c197525542bc8e386246a8c3375f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:10.981643 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4215c197525542bc8e386246a8c3375f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:10.988940 24959 tablet_bootstrap.cc:492] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db: Bootstrap starting.
I20260620 14:51:10.990794 24960 tablet_bootstrap.cc:492] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51: Bootstrap starting.
I20260620 14:51:10.992321 24959 tablet_bootstrap.cc:654] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:10.992941 24960 tablet_bootstrap.cc:654] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:10.993471 24959 log.cc:824] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:10.993904 24960 log.cc:824] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:10.994279 24961 tablet_bootstrap.cc:492] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22: Bootstrap starting.
I20260620 14:51:10.996368 24960 tablet_bootstrap.cc:492] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51: No bootstrap required, opened a new log
I20260620 14:51:10.996366 24959 tablet_bootstrap.cc:492] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db: No bootstrap required, opened a new log
I20260620 14:51:10.996618 24959 ts_tablet_manager.cc:1397] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 14:51:10.996640 24960 ts_tablet_manager.cc:1397] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:51:10.996690 24961 tablet_bootstrap.cc:654] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:10.998147 24961 log.cc:824] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:11.001068 24959 raft_consensus.cc:348] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } }
I20260620 14:51:11.001312 24961 tablet_bootstrap.cc:492] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22: No bootstrap required, opened a new log
I20260620 14:51:11.001384 24959 raft_consensus.cc:374] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:11.001443 24959 raft_consensus.cc:729] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4012070e7f0048a5b20486063abc30db, State: Initialized, Role: FOLLOWER
I20260620 14:51:11.001549 24961 ts_tablet_manager.cc:1397] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 14:51:11.001693 24960 raft_consensus.cc:348] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } }
I20260620 14:51:11.002115 24960 raft_consensus.cc:374] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:11.002063 24959 consensus_queue.cc:260] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } }
I20260620 14:51:11.002184 24960 raft_consensus.cc:729] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92be6e49c874c1298cade013b065f51, State: Initialized, Role: FOLLOWER
I20260620 14:51:11.002784 24960 consensus_queue.cc:260] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } }
I20260620 14:51:11.002923 24808 heartbeater.cc:503] Master 127.8.221.254:38213 was elected leader, sending a full tablet report...
I20260620 14:51:11.003185 24959 ts_tablet_manager.cc:1428] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260620 14:51:11.003571 24674 heartbeater.cc:503] Master 127.8.221.254:38213 was elected leader, sending a full tablet report...
I20260620 14:51:11.005366 24960 ts_tablet_manager.cc:1428] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260620 14:51:11.007289 24961 raft_consensus.cc:348] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } }
I20260620 14:51:11.007619 24961 raft_consensus.cc:374] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:11.007688 24961 raft_consensus.cc:729] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bfab1be9f36478bbf52b29235163b22, State: Initialized, Role: FOLLOWER
I20260620 14:51:11.008164 24961 consensus_queue.cc:260] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } }
I20260620 14:51:11.008853 24942 heartbeater.cc:503] Master 127.8.221.254:38213 was elected leader, sending a full tablet report...
I20260620 14:51:11.009248 24961 ts_tablet_manager.cc:1428] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260620 14:51:11.058475 24965 raft_consensus.cc:482] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:11.058816 24965 raft_consensus.cc:504] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } }
I20260620 14:51:11.060351 24965 leader_election.cc:288] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b92be6e49c874c1298cade013b065f51 (127.8.221.193:45499), 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049)
I20260620 14:51:11.064479 24810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:11.060679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:45499 (local address 127.8.221.194:41609)
0620 14:51:11.060831 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:11.060852 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:11.060979 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:11.061351 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:11.061356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:11.061382 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:11.061498 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:11.061504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:11.062357 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:11.062361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:11.063238 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:11.063248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:11.063965 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:11.063969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:11.064090 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:11.064109 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:51:11.064236 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 14:51:11.064327 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 14:51:11.064479 24955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:11.060850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:41609 (local address 127.8.221.193:45499)
0620 14:51:11.061021 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:11.061028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:11.061051 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:51:11.061130 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:11.061134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:11.061193 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:11.061296 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:11.061626 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:11.062231 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:11.063380 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:11.063830 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:11.063973 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:11.064031 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:11.064262 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 14:51:11.064339 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:51:11.064837 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:11.061112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:39577 (local address 127.8.221.195:40049)
0620 14:51:11.061273 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:11.061278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:11.061411 (+ 133us) server_negotiation.cc:407] Connection header received
0620 14:51:11.061571 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:11.061574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:11.061628 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:11.061721 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:11.062055 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:11.062604 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:11.063691 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:11.064112 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:11.064303 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:11.064366 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:11.064577 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:51:11.064686 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:51:11.064849 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:11.060850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:40049 (local address 127.8.221.194:39577)
0620 14:51:11.061298 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:11.061313 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:11.061447 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:11.061732 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:11.061735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:11.061750 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:11.061927 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:11.061933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:11.062734 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:11.062739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:11.063549 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:11.063557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:11.064308 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:11.064312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:11.064418 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:11.064449 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:51:11.064557 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:51:11.064643 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":178,"threads_started":1}
I20260620 14:51:11.065289 24628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4215c197525542bc8e386246a8c3375f" candidate_uuid: "4012070e7f0048a5b20486063abc30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92be6e49c874c1298cade013b065f51" is_pre_election: true
I20260620 14:51:11.065685 24628 raft_consensus.cc:2454] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4012070e7f0048a5b20486063abc30db in term 0.
I20260620 14:51:11.065871 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4215c197525542bc8e386246a8c3375f" candidate_uuid: "4012070e7f0048a5b20486063abc30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bfab1be9f36478bbf52b29235163b22" is_pre_election: true
I20260620 14:51:11.066295 24896 raft_consensus.cc:2454] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4012070e7f0048a5b20486063abc30db in term 0.
I20260620 14:51:11.066391 24698 leader_election.cc:302] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4012070e7f0048a5b20486063abc30db, b92be6e49c874c1298cade013b065f51; no voters:
I20260620 14:51:11.066823 24965 raft_consensus.cc:2781] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:11.066928 24965 raft_consensus.cc:482] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:11.066960 24965 raft_consensus.cc:3037] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:11.068427 24965 raft_consensus.cc:504] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } }
I20260620 14:51:11.068948 24965 leader_election.cc:288] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [CANDIDATE]: Term 1 election: Requested vote from peers b92be6e49c874c1298cade013b065f51 (127.8.221.193:45499), 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049)
I20260620 14:51:11.069437 24628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4215c197525542bc8e386246a8c3375f" candidate_uuid: "4012070e7f0048a5b20486063abc30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92be6e49c874c1298cade013b065f51"
I20260620 14:51:11.069480 24896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4215c197525542bc8e386246a8c3375f" candidate_uuid: "4012070e7f0048a5b20486063abc30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bfab1be9f36478bbf52b29235163b22"
I20260620 14:51:11.069628 24628 raft_consensus.cc:3037] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:11.069648 24896 raft_consensus.cc:3037] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:11.071341 24896 raft_consensus.cc:2454] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4012070e7f0048a5b20486063abc30db in term 1.
I20260620 14:51:11.071365 24628 raft_consensus.cc:2454] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4012070e7f0048a5b20486063abc30db in term 1.
I20260620 14:51:11.071780 24695 leader_election.cc:302] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bfab1be9f36478bbf52b29235163b22, 4012070e7f0048a5b20486063abc30db; no voters:
I20260620 14:51:11.072108 24965 raft_consensus.cc:2781] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:11.072336 24965 raft_consensus.cc:686] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [term 1 LEADER]: Becoming Leader. State: Replica: 4012070e7f0048a5b20486063abc30db, State: Running, Role: LEADER
I20260620 14:51:11.072733 24965 consensus_queue.cc:237] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } }
I20260620 14:51:11.076586 24483 catalog_manager.cc:5495] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db reported cstate change: term changed from 0 to 1, leader changed from <none> to 4012070e7f0048a5b20486063abc30db (127.8.221.194). New cstate: current_term: 1 leader_uuid: "4012070e7f0048a5b20486063abc30db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } health_report { overall_health: UNKNOWN } } }
I20260620 14:51:11.106635 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:11.102421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.8.221.194:39433)
0620 14:51:11.102602 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:11.102610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:11.102628 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:51:11.102752 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:11.102755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:11.102822 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:11.102958 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:11.103239 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:11.103806 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:11.104779 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:11.105008 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:11.105145 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:11.105522 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:11.106176 (+ 654us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:11.106331 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:11.106368 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 14:51:11.106440 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:51:11.114181 24896 raft_consensus.cc:1261] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22 [term 1 FOLLOWER]: Refusing update from remote peer 4012070e7f0048a5b20486063abc30db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:11.114185 24628 raft_consensus.cc:1261] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Refusing update from remote peer 4012070e7f0048a5b20486063abc30db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:11.115159 24965 consensus_queue.cc:1035] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:11.115541 24969 consensus_queue.cc:1035] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:11.127951 24974 mvcc.cc:205] Tried to move back new op lower bound from 7298937123274260480 to 7298937123117301760. Current Snapshot: MvccSnapshot[applied={T|T < 7298937123274260480}]
I20260620 14:51:11.129909 24973 mvcc.cc:205] Tried to move back new op lower bound from 7298937123274260480 to 7298937123117301760. Current Snapshot: MvccSnapshot[applied={T|T < 7298937123274260480}]
I20260620 14:51:11.137454 24482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:11.137694 24482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:11.141091 24482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c: Sending DeleteTablet for 3 replicas of tablet 4215c197525542bc8e386246a8c3375f
I20260620 14:51:11.142246 24742 tablet_service.cc:1514] Processing DeleteTablet for tablet 4215c197525542bc8e386246a8c3375f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:11 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:55108
I20260620 14:51:11.142449 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 24545
I20260620 14:51:11.142474 24608 tablet_service.cc:1514] Processing DeleteTablet for tablet 4215c197525542bc8e386246a8c3375f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:11 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:58008
I20260620 14:51:11.143215 24876 tablet_service.cc:1514] Processing DeleteTablet for tablet 4215c197525542bc8e386246a8c3375f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:11 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:46254
I20260620 14:51:11.143776 24983 tablet_replica.cc:330] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22: stopping tablet replica
I20260620 14:51:11.144237 24983 raft_consensus.cc:2229] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:11.144668 24982 tablet_replica.cc:330] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db: stopping tablet replica
I20260620 14:51:11.144690 24983 raft_consensus.cc:2258] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:11.145644 24982 raft_consensus.cc:2229] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:51:11.146201 24982 raft_consensus.cc:2258] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:11.146641 24983 ts_tablet_manager.cc:1905] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:11.149621 24983 ts_tablet_manager.cc:1918] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:11.149731 24983 log.cc:1196] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/4215c197525542bc8e386246a8c3375f
I20260620 14:51:11.150115 24983 ts_tablet_manager.cc:1939] T 4215c197525542bc8e386246a8c3375f P 1bfab1be9f36478bbf52b29235163b22: Deleting consensus metadata
I20260620 14:51:11.150612 24982 ts_tablet_manager.cc:1905] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:11.151474 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 24679
W20260620 14:51:11.151585 24470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.221.193:45499 (error 108)
I20260620 14:51:11.152891 24467 catalog_manager.cc:4841] TS 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049): tablet 4215c197525542bc8e386246a8c3375f (table test-table [id=c096fecd7f7047eab5362216eba325ef]) successfully deleted
I20260620 14:51:11.153684 24956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:11.153301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:45499 (local address 127.0.0.1:58010)
0620 14:51:11.153526 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:11.153592 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.193:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260620 14:51:11.153993 24470 catalog_manager.cc:4584] TS b92be6e49c874c1298cade013b065f51 (127.8.221.193:45499): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4215c197525542bc8e386246a8c3375f: Network error: Client connection negotiation failed: client connection to 127.8.221.193:45499: connect: Connection refused (error 111)
I20260620 14:51:11.159641 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 24813
I20260620 14:51:11.159744 24956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:11.159271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:39433 (local address 127.0.0.1:55136)
0620 14:51:11.159544 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:11.159618 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.194:39433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 14:51:11.166908 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 24451
I20260620 14:51:31.176303 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38213
--webserver_interface=127.8.221.254
--webserver_port=46171
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:38213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:31.309047 24985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:31.309396 24985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:31.309441 24985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:31.313891 24985 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:51:31.313957 24985 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:51:31.313974 24985 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:51:31.313989 24985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:31.314009 24985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:31.314028 24985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:31.314044 24985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:31.314060 24985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:31.320091 24985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:38213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=46171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:51:31.321383 24985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:31.322602 24985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:31.338726 24991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:31.338768 24992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:31.338824 24994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:31.339224 24985 server_base.cc:1034] running on GCE node
I20260620 14:51:31.340049 24985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:31.341590 24985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:31.342763 24985 hybrid_clock.cc:648] HybridClock initialized: now 1781967091342718 us; error 61 us; skew 500 ppm
Jun 20 14:51:31 dist-test-slave-65tz krb5kdc[24438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:31.346127 24985 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:51:31.347255 24985 webserver.cc:460] Webserver started at http://127.8.221.254:46171/ using document root <none> and password file <none>
I20260620 14:51:31.347911 24985 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:31.347990 24985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:31.352690 24985 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:51:31.355480 25001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:31.357043 24985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:51:31.357245 24985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "cdc7bbdfb08641b794835c384921cd7c"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "4ef63526ad5903b7a719f414cc74974c"
server_key_iv: "5e5d090e9e7b4b56596b9d9cb044b262"
server_key_version: "encryptionkey@0"
I20260620 14:51:31.358047 24985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:31.376286 24985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:31.380264 24985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:31.380509 24985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:31.390005 24985 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:38213
I20260620 14:51:31.390369 25053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:38213 every 8 connection(s)
I20260620 14:51:31.391387 24985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:31.394905 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 24985
I20260620 14:51:31.395462 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:45499
--local_ip_for_outbound_sockets=127.8.221.193
--tserver_master_addrs=127.8.221.254:38213
--webserver_port=44527
--webserver_interface=127.8.221.193
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:51:31.396898 25054 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:51:31.398789 25054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c: Bootstrap starting.
I20260620 14:51:31.416436 25054 log.cc:824] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:31.425266 25054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:51:31.425693 25054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c: Bootstrap complete.
I20260620 14:51:31.429931 25054 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7bbdfb08641b794835c384921cd7c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38213 } }
I20260620 14:51:31.430361 25054 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc7bbdfb08641b794835c384921cd7c, State: Initialized, Role: FOLLOWER
I20260620 14:51:31.430884 25054 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7bbdfb08641b794835c384921cd7c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38213 } }
I20260620 14:51:31.431001 25054 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:31.431095 25054 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:31.431221 25054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:51:31.433269 25054 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7bbdfb08641b794835c384921cd7c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38213 } }
I20260620 14:51:31.433629 25054 leader_election.cc:302] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdc7bbdfb08641b794835c384921cd7c; no voters:
I20260620 14:51:31.434011 25054 leader_election.cc:288] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:51:31.434095 25058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:51:31.434303 25058 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 2 LEADER]: Becoming Leader. State: Replica: cdc7bbdfb08641b794835c384921cd7c, State: Running, Role: LEADER
I20260620 14:51:31.434617 25058 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7bbdfb08641b794835c384921cd7c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38213 } }
I20260620 14:51:31.435352 25054 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:31.436014 25059 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cdc7bbdfb08641b794835c384921cd7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7bbdfb08641b794835c384921cd7c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38213 } } }
I20260620 14:51:31.436144 25059 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:31.436479 25060 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdc7bbdfb08641b794835c384921cd7c. Latest consensus state: current_term: 2 leader_uuid: "cdc7bbdfb08641b794835c384921cd7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7bbdfb08641b794835c384921cd7c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38213 } } }
I20260620 14:51:31.436607 25060 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:31.437417 25063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:31.443296 25063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:31.443928 25063 catalog_manager.cc:1259] Loaded cluster ID: 2ff1698fa3c04f44ad6210cbe10e7a60
I20260620 14:51:31.443989 25063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:31.446849 25063 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:31.447913 25063 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c: Loaded TSK: 0
I20260620 14:51:31.472401 25063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c: Generated new TSK 1
I20260620 14:51:31.472810 25063 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:51:31.535362 25056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:31.535684 25056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:31.535759 25056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:31.540445 25056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:31.540509 25056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:31.540608 25056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:51:31.546386 25056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:45499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=44527
--tserver_master_addrs=127.8.221.254:38213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:51:31.547621 25056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:31.548842 25056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:31.560612 25084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:31.560580 25082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:31.560689 25081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:31.561023 25056 server_base.cc:1034] running on GCE node
I20260620 14:51:31.561595 25056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:31.562469 25056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:31.563676 25056 hybrid_clock.cc:648] HybridClock initialized: now 1781967091563607 us; error 115 us; skew 500 ppm
Jun 20 14:51:31 dist-test-slave-65tz krb5kdc[24438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:31.566910 25056 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:51:31.568059 25056 webserver.cc:460] Webserver started at http://127.8.221.193:44527/ using document root <none> and password file <none>
I20260620 14:51:31.568750 25056 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:31.568862 25056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:31.574095 25056 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:51:31.576798 25091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:31.578562 25056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:51:31.578742 25056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "b92be6e49c874c1298cade013b065f51"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "3439e63d12178a49e12b492223efeeef"
server_key_iv: "c625ce12042ccb186525bbc77542df8d"
server_key_version: "encryptionkey@0"
I20260620 14:51:31.579386 25056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:31.599609 25056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:31.602991 25056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:31.603214 25056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:31.603904 25056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:31.605229 25098 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:51:31.609854 25056 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:51:31.609975 25056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260620 14:51:31.610081 25056 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:51:31.612257 25056 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:51:31.612336 25056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 14:51:31.612759 25098 tablet_bootstrap.cc:492] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51: Bootstrap starting.
I20260620 14:51:31.625015 25056 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:45499
I20260620 14:51:31.625151 25205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:45499 every 8 connection(s)
I20260620 14:51:31.626323 25056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:51:31.633127 25098 log.cc:824] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:31.634459 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 25056
I20260620 14:51:31.635023 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:39433
--local_ip_for_outbound_sockets=127.8.221.194
--tserver_master_addrs=127.8.221.254:38213
--webserver_port=37501
--webserver_interface=127.8.221.194
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:51:31 dist-test-slave-65tz krb5kdc[24438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:51:31.643110 25098 tablet_bootstrap.cc:492] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:51:31.643636 25098 tablet_bootstrap.cc:492] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51: Bootstrap complete.
I20260620 14:51:31.644891 25098 ts_tablet_manager.cc:1397] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51: Time spent bootstrapping tablet: real 0.032s user 0.028s sys 0.000s
I20260620 14:51:31.647941 25098 raft_consensus.cc:348] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } }
I20260620 14:51:31.648511 25098 raft_consensus.cc:729] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92be6e49c874c1298cade013b065f51, State: Initialized, Role: FOLLOWER
I20260620 14:51:31.648782 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.628850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:42895)
0620 14:51:31.630317 (+ 1467us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:31.630389 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:31.631597 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:31.632988 (+ 1391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:31.632997 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:31.633511 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:31.634209 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.634229 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.636099 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.636102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.637465 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.637480 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.638635 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.638642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.638887 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.639697 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:31.639722 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:31.641897 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:31.644893 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.644900 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.644917 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.645299 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.645773 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.645777 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.645779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.645984 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.646838 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:31.646844 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:31.647150 (+ 306us) client_negotiation.cc:769] Sending connection context
0620 14:51:31.647351 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 14:51:31.647735 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1204,"thread_start_us":186,"threads_started":1}
I20260620 14:51:31.649048 25098 consensus_queue.cc:260] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } }
I20260620 14:51:31.649682 25208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.628839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:42895 (local address 127.8.221.254:38213)
0620 14:51:31.629263 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.629282 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.630571 (+ 1289us) server_negotiation.cc:407] Connection header received
0620 14:51:31.632058 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.632072 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.632445 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.632876 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:31.634477 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.635926 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.637626 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.638427 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.638621 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.642161 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:31.642172 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:31.642197 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:31.642246 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:31.644736 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.645466 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.645476 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.645484 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.645651 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.646120 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.646126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.646130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.646696 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:31.646859 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:31.647558 (+ 699us) server_negotiation.cc:299] Negotiation successful
0620 14:51:31.647928 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":142,"threads_started":1}
I20260620 14:51:31.650187 25098 ts_tablet_manager.cc:1428] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260620 14:51:31.652297 25206 heartbeater.cc:347] Connected to a master server at 127.8.221.254:38213
I20260620 14:51:31.652513 25206 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:31.653007 25206 heartbeater.cc:511] Master 127.8.221.254:38213 requested a full tablet report, sending...
I20260620 14:51:31.655831 25018 ts_manager.cc:194] Registered new tserver with Master: b92be6e49c874c1298cade013b065f51 (127.8.221.193:45499)
I20260620 14:51:31.659818 25018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:42895
I20260620 14:51:31.662875 25206 heartbeater.cc:503] Master 127.8.221.254:38213 was elected leader, sending a full tablet report...
Jun 20 14:51:31 dist-test-slave-65tz krb5kdc[24438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for kudu/127.8.221.193@KRBTEST.COM
I20260620 14:51:31.671381 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.659223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59464 (local address 127.8.221.193:45499)
0620 14:51:31.659967 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.659981 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.660008 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:51:31.660091 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.660097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.660312 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.660472 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:31.661365 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.662617 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.664032 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.664588 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.664841 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.667490 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:31.667506 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:31.667518 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:31.667540 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:31.669250 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.669777 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.669782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.669784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.669919 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.670223 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.670227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.670229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.670493 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:31.670639 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:31.670999 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 14:51:31.671178 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":627,"thread_start_us":157,"threads_started":1}
I20260620 14:51:31.671381 25213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.659094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:45499 (local address 127.0.0.1:59464)
0620 14:51:31.659616 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:31.659675 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:31.659835 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:31.660507 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:31.660513 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:31.661058 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:31.661216 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.661224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.662757 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.662761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.663884 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.663895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.664874 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.664878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.664989 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.665652 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:31.665667 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:31.667343 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:31.669375 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.669386 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.669392 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.669644 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.670023 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.670026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.670028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.670113 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.670619 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:31.670626 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:31.670764 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 14:51:31.670921 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 14:51:31.671165 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":139,"threads_started":1}
I20260620 14:51:31.672809 25140 tablet_service.cc:1514] Processing DeleteTablet for tablet 4215c197525542bc8e386246a8c3375f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:11 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:59464
I20260620 14:51:31.673928 25215 tablet_replica.cc:330] stopping tablet replica
I20260620 14:51:31.674304 25215 raft_consensus.cc:2229] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:31.674598 25215 raft_consensus.cc:2258] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:31.675822 25215 ts_tablet_manager.cc:1905] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:31.680528 25215 ts_tablet_manager.cc:1918] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:31.680637 25215 log.cc:1196] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal/wals/4215c197525542bc8e386246a8c3375f
I20260620 14:51:31.680962 25215 ts_tablet_manager.cc:1939] T 4215c197525542bc8e386246a8c3375f P b92be6e49c874c1298cade013b065f51: Deleting consensus metadata
I20260620 14:51:31.683238 25005 catalog_manager.cc:4841] TS b92be6e49c874c1298cade013b065f51 (127.8.221.193:45499): tablet 4215c197525542bc8e386246a8c3375f (table test-table [id=c096fecd7f7047eab5362216eba325ef]) successfully deleted
W20260620 14:51:31.763656 25211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:31.763924 25211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:31.763969 25211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:31.768061 25211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:31.768129 25211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:31.768229 25211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:51:31.773669 25211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:39433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=37501
--tserver_master_addrs=127.8.221.254:38213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:51:31.774824 25211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:31.775869 25211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:31.786231 25223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:31.786231 25220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:31.786427 25211 server_base.cc:1034] running on GCE node
W20260620 14:51:31.786221 25221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:31.787019 25211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:31.787781 25211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:31.789000 25211 hybrid_clock.cc:648] HybridClock initialized: now 1781967091788976 us; error 40 us; skew 500 ppm
Jun 20 14:51:31 dist-test-slave-65tz krb5kdc[24438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:31.792043 25211 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:51:31.793005 25211 webserver.cc:460] Webserver started at http://127.8.221.194:37501/ using document root <none> and password file <none>
I20260620 14:51:31.793641 25211 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:31.793752 25211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:31.797767 25211 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:51:31.800238 25230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:31.801662 25211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:51:31.801867 25211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "4012070e7f0048a5b20486063abc30db"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "8ca0b741c9b79cf9721b330927db9d87"
server_key_iv: "a0de6a1873627d1bd21f05e28ef66043"
server_key_version: "encryptionkey@0"
I20260620 14:51:31.802460 25211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:31.818650 25211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:31.822140 25211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:31.822348 25211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:31.822956 25211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:31.824117 25237 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:51:31.829587 25211 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:51:31.829681 25211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 14:51:31.829788 25211 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:51:31.831689 25211 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:51:31.831761 25211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 14:51:31.832140 25237 tablet_bootstrap.cc:492] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db: Bootstrap starting.
I20260620 14:51:31.840118 25237 log.cc:824] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:31.843246 25211 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:39433
I20260620 14:51:31.843333 25345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:39433 every 8 connection(s)
I20260620 14:51:31.844249 25211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:51:31.850502 25237 tablet_bootstrap.cc:492] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:51:31.850955 25237 tablet_bootstrap.cc:492] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db: Bootstrap complete.
I20260620 14:51:31.851388 25237 ts_tablet_manager.cc:1397] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.000s
I20260620 14:51:31.854139 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 25211
I20260620 14:51:31.854683 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:40049
--local_ip_for_outbound_sockets=127.8.221.195
--tserver_master_addrs=127.8.221.254:38213
--webserver_port=44845
--webserver_interface=127.8.221.195
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:51:31.854797 25237 raft_consensus.cc:348] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } }
Jun 20 14:51:31 dist-test-slave-65tz krb5kdc[24438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:51:31.855489 25237 raft_consensus.cc:729] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4012070e7f0048a5b20486063abc30db, State: Initialized, Role: FOLLOWER
I20260620 14:51:31.856993 25237 consensus_queue.cc:260] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } }
I20260620 14:51:31.858249 25237 ts_tablet_manager.cc:1428] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 14:51:31.861299 25208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.846487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:50267 (local address 127.8.221.254:38213)
0620 14:51:31.846680 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.846687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.847364 (+ 677us) server_negotiation.cc:407] Connection header received
0620 14:51:31.848589 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.848592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.848651 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.848741 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:31.850326 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.850938 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.852666 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.852903 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.853021 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.857164 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:31.857168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:31.857172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:31.857211 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:31.858945 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.859624 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.859627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.859628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.859706 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.860121 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.860125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.860127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.860347 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:31.860460 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:31.860957 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 14:51:31.861113 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:51:31.862354 25348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.846614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:50267)
0620 14:51:31.847144 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:31.847199 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:31.848315 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:31.848930 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:31.848939 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:31.849426 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:31.850146 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.850169 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.851128 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.851132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.852504 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.852514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.853057 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.853063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.853296 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.854709 (+ 1413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:31.854772 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:31.856900 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:31.859083 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.859092 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.859110 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.859496 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.859849 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.859853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.859855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.860014 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.860458 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:31.860465 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:31.860721 (+ 256us) client_negotiation.cc:769] Sending connection context
0620 14:51:31.860918 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 14:51:31.861287 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":121,"threads_started":1}
I20260620 14:51:31.863682 25346 heartbeater.cc:347] Connected to a master server at 127.8.221.254:38213
I20260620 14:51:31.863976 25346 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:31.864537 25346 heartbeater.cc:511] Master 127.8.221.254:38213 requested a full tablet report, sending...
I20260620 14:51:31.866739 25018 ts_manager.cc:194] Registered new tserver with Master: 4012070e7f0048a5b20486063abc30db (127.8.221.194:39433)
I20260620 14:51:31.868186 25018 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:50267
I20260620 14:51:31.871009 25346 heartbeater.cc:503] Master 127.8.221.254:38213 was elected leader, sending a full tablet report...
Jun 20 14:51:31 dist-test-slave-65tz krb5kdc[24438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for kudu/127.8.221.194@KRBTEST.COM
I20260620 14:51:31.878475 25213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.867850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:39433 (local address 127.0.0.1:54368)
0620 14:51:31.867983 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:31.867996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:31.868067 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:31.869078 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:31.869081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:31.869308 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:31.869406 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.869412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.870519 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.870525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.871105 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.871116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.871855 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.871860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.872011 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.872600 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:31.872614 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:31.874288 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:31.876388 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.876391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.876394 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.876671 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.877064 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.877067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.877069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.877164 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.877773 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:31.877777 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:31.877924 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 14:51:31.878091 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 14:51:31.878278 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 14:51:31.878547 25351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.868201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54368 (local address 127.8.221.194:39433)
0620 14:51:31.868586 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.868598 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.868619 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:51:31.868685 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.868691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.868943 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.869099 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:31.869516 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.870398 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.871245 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.871723 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.871832 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.874450 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:31.874459 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:31.874473 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:31.874495 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:31.876262 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.876835 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.876840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.876842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.876947 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.877341 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.877344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.877345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.877635 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:31.877762 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:31.878160 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 14:51:31.878353 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":278,"thread_start_us":184,"threads_started":1}
I20260620 14:51:31.879877 25279 tablet_service.cc:1514] Processing DeleteTablet for tablet 4215c197525542bc8e386246a8c3375f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:11 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:54368
I20260620 14:51:31.881000 25352 tablet_replica.cc:330] stopping tablet replica
I20260620 14:51:31.881367 25352 raft_consensus.cc:2229] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:31.881667 25352 raft_consensus.cc:2258] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:31.882881 25352 ts_tablet_manager.cc:1905] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:31.887753 25352 ts_tablet_manager.cc:1918] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:31.887868 25352 log.cc:1196] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/4215c197525542bc8e386246a8c3375f
I20260620 14:51:31.888209 25352 ts_tablet_manager.cc:1939] T 4215c197525542bc8e386246a8c3375f P 4012070e7f0048a5b20486063abc30db: Deleting consensus metadata
I20260620 14:51:31.890123 25004 catalog_manager.cc:4841] TS 4012070e7f0048a5b20486063abc30db (127.8.221.194:39433): tablet 4215c197525542bc8e386246a8c3375f (table test-table [id=c096fecd7f7047eab5362216eba325ef]) successfully deleted
W20260620 14:51:31.974689 25349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:31.974997 25349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:31.975070 25349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:31.979141 25349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:31.979225 25349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:31.979314 25349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:51:31.984865 25349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:40049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=44845
--tserver_master_addrs=127.8.221.254:38213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:51:31.986060 25349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:31.987140 25349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:31.997371 25357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:31.997371 25358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:31.997411 25360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:31.997884 25349 server_base.cc:1034] running on GCE node
I20260620 14:51:31.998404 25349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:31.999261 25349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:32.000465 25349 hybrid_clock.cc:648] HybridClock initialized: now 1781967092000452 us; error 33 us; skew 500 ppm
Jun 20 14:51:31 dist-test-slave-65tz krb5kdc[24438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:32.003623 25349 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:51:32.004647 25349 webserver.cc:460] Webserver started at http://127.8.221.195:44845/ using document root <none> and password file <none>
I20260620 14:51:32.005272 25349 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:32.005355 25349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:32.009995 25349 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:51:32.012643 25367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:32.014335 25349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:51:32.014457 25349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "1bfab1be9f36478bbf52b29235163b22"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "fb59d68d07fa8c5e512c568485f77a9a"
server_key_iv: "a0eea83d54321860fa484a4eeb9c542b"
server_key_version: "encryptionkey@0"
I20260620 14:51:32.015118 25349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:32.049603 25349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:32.053099 25349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:32.053300 25349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:32.053990 25349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:32.055013 25349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:32.055064 25349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:32.055142 25349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:32.055186 25349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:32.066440 25349 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:40049
I20260620 14:51:32.066541 25480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:40049 every 8 connection(s)
I20260620 14:51:32.067691 25349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:51:32.074890 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 25349
Jun 20 14:51:32 dist-test-slave-65tz krb5kdc[24438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:51:32.084214 25208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.070074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:49375 (local address 127.8.221.254:38213)
0620 14:51:32.070261 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:32.070265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:32.071487 (+ 1222us) server_negotiation.cc:407] Connection header received
0620 14:51:32.072407 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:32.072410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:32.072469 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:32.072581 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:32.073897 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.074461 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.076085 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.076426 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.076536 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.079798 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:32.079801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:32.079804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:32.079831 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:32.081655 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:32.082412 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:32.082416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:32.082418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:32.082506 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:32.082907 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:32.082911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:32.082940 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:32.083110 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:32.083231 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:32.083884 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 14:51:32.084024 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":79}
I20260620 14:51:32.085062 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.070491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.195:49375)
0620 14:51:32.071291 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:32.071339 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:32.072187 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:32.072762 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:32.072769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:32.073171 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:32.073692 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.073705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.074610 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.074616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.075915 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.075927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.076599 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.076603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.076794 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.077472 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:32.077497 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:32.079574 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:32.081781 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:32.081791 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:32.081830 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:32.082199 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:32.082605 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:32.082609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:32.082611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:32.082795 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:32.083240 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:32.083248 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:32.083543 (+ 295us) client_negotiation.cc:769] Sending connection context
0620 14:51:32.083754 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 14:51:32.084109 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":140,"threads_started":1}
I20260620 14:51:32.085412 25484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.075675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46798 (local address 127.8.221.254:38213)
0620 14:51:32.075989 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:32.075997 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:32.076209 (+ 212us) server_negotiation.cc:407] Connection header received
0620 14:51:32.076354 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:32.076357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:32.076413 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:32.076518 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:32.078124 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.078651 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.079691 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.079950 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.080030 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.081685 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:32.081688 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:32.081691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:32.081737 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:32.083513 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:32.084042 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:32.084047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:32.084051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:32.084155 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:32.084439 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:32.084444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:32.084446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:32.084695 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:32.084919 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:32.085068 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 14:51:32.085199 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":216,"thread_start_us":101,"threads_started":1}
I20260620 14:51:32.086449 25481 heartbeater.cc:347] Connected to a master server at 127.8.221.254:38213
I20260620 14:51:32.086725 25481 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:32.087301 25481 heartbeater.cc:511] Master 127.8.221.254:38213 requested a full tablet report, sending...
I20260620 14:51:32.088362 25017 ts_manager.cc:194] Registered new tserver with Master: 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049)
I20260620 14:51:32.088963 25017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:49375
I20260620 14:51:32.099577 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:51:32.104179 25484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.100497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46808 (local address 127.8.221.254:38213)
0620 14:51:32.100671 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:32.100675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:32.100887 (+ 212us) server_negotiation.cc:407] Connection header received
0620 14:51:32.101039 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:32.101042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:32.101173 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:32.101273 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:32.101497 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.102155 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.103034 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.103261 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.103339 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.103637 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:32.103818 (+ 181us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 14:51:32.104059 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46808: token expired
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 14:51:32.104250 25484 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46808: token expired
W20260620 14:51:32.104605 24948 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.221.254:38213)
I20260620 14:51:32.111891 25484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.105006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46814 (local address 127.8.221.254:38213)
0620 14:51:32.105126 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:32.105130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:32.105152 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:51:32.105224 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:32.105226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:32.105261 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:32.105332 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:32.106048 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.106514 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.107324 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.107520 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.107604 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.108872 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:32.108874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:32.108876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:32.108897 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:32.110481 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:32.110905 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:32.110907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:32.110908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:32.110979 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:32.111235 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:32.111237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:32.111238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:32.111398 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:32.111508 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:32.111682 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 14:51:32.111770 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:51:32.114517 25017 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:51:32.117915 25017 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:32.126749 25279 tablet_service.cc:1467] Processing CreateTablet for tablet 35f9a2c48e66416182d488e2e8460e29 (DEFAULT_TABLE table=test-table [id=390f00dbf711494d9fbd1aa2ce0d694b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:32.126971 25140 tablet_service.cc:1467] Processing CreateTablet for tablet 35f9a2c48e66416182d488e2e8460e29 (DEFAULT_TABLE table=test-table [id=390f00dbf711494d9fbd1aa2ce0d694b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:32.127444 25279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35f9a2c48e66416182d488e2e8460e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:32.127748 25140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35f9a2c48e66416182d488e2e8460e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:32.130401 25237 tablet_bootstrap.cc:492] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db: Bootstrap starting.
I20260620 14:51:32.130834 25098 tablet_bootstrap.cc:492] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51: Bootstrap starting.
I20260620 14:51:32.131471 25237 tablet_bootstrap.cc:654] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:32.131744 25098 tablet_bootstrap.cc:654] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:32.132745 25213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.126037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:40049 (local address 127.0.0.1:58166)
0620 14:51:32.126399 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:32.126416 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:32.126499 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:32.127416 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:32.127421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:32.127444 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:32.127581 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.127588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.129547 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.129551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.130889 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.130901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.132169 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.132175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.132295 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.132360 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 14:51:32.132463 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 14:51:32.132565 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266}
I20260620 14:51:32.132812 25490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.126248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.8.221.195:40049)
0620 14:51:32.126848 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:32.126864 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:32.126892 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:51:32.126978 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:32.126984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:32.127213 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:32.127395 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:32.128082 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.129399 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.131044 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.132007 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.132135 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.132318 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:32.132513 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 14:51:32.132622 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":221,"threads_started":1}
I20260620 14:51:32.134728 25098 tablet_bootstrap.cc:492] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51: No bootstrap required, opened a new log
I20260620 14:51:32.134846 25098 ts_tablet_manager.cc:1397] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 14:51:32.135064 25237 tablet_bootstrap.cc:492] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db: No bootstrap required, opened a new log
I20260620 14:51:32.135165 25237 ts_tablet_manager.cc:1397] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 14:51:32.135416 25098 raft_consensus.cc:348] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } }
I20260620 14:51:32.135563 25098 raft_consensus.cc:374] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:32.135613 25098 raft_consensus.cc:729] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92be6e49c874c1298cade013b065f51, State: Initialized, Role: FOLLOWER
I20260620 14:51:32.135625 25237 raft_consensus.cc:348] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } }
I20260620 14:51:32.135756 25237 raft_consensus.cc:374] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:32.135792 25237 raft_consensus.cc:729] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4012070e7f0048a5b20486063abc30db, State: Initialized, Role: FOLLOWER
I20260620 14:51:32.135936 25237 consensus_queue.cc:260] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } }
I20260620 14:51:32.135949 25098 consensus_queue.cc:260] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } }
I20260620 14:51:32.136197 25237 ts_tablet_manager.cc:1428] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:32.136261 25098 ts_tablet_manager.cc:1428] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:32.136642 25415 tablet_service.cc:1467] Processing CreateTablet for tablet 35f9a2c48e66416182d488e2e8460e29 (DEFAULT_TABLE table=test-table [id=390f00dbf711494d9fbd1aa2ce0d694b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:32.138057 25415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35f9a2c48e66416182d488e2e8460e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:32.144989 25350 raft_consensus.cc:482] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:32.145253 25350 raft_consensus.cc:504] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } }
I20260620 14:51:32.146207 25493 tablet_bootstrap.cc:492] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22: Bootstrap starting.
I20260620 14:51:32.146780 25350 leader_election.cc:288] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049), b92be6e49c874c1298cade013b065f51 (127.8.221.193:45499)
I20260620 14:51:32.150063 25493 tablet_bootstrap.cc:654] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:32.151171 25493 log.cc:824] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:32.152252 25348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.147004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:40049 (local address 127.8.221.194:59531)
0620 14:51:32.147260 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:32.147284 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:32.147414 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:32.148483 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:32.148486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:32.148504 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:32.148618 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.148624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.149578 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.149584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.150773 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.150784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.151747 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.151753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.151858 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.151900 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:51:32.151986 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:51:32.152071 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":34}
I20260620 14:51:32.152320 25490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.148125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:59531 (local address 127.8.221.195:40049)
0620 14:51:32.148249 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:32.148254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:32.148267 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:51:32.148317 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:32.148320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:32.148365 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:32.148447 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:32.148814 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.149446 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.150902 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.151489 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.151617 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.151678 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:32.152041 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 14:51:32.152165 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:51:32.152467 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.147004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:45499 (local address 127.8.221.194:39833)
0620 14:51:32.147469 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:32.147482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:32.147571 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:32.147871 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:32.147873 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:32.147885 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:32.148053 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.148062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.149213 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.149217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.150800 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.150812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.152043 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.152047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.152139 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.152160 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:51:32.152237 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:51:32.152320 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":183,"thread_start_us":149,"threads_started":1}
I20260620 14:51:32.152537 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.147100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:39833 (local address 127.8.221.193:45499)
0620 14:51:32.147617 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:32.147622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:32.147637 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:51:32.147688 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:32.147691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:32.147753 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:32.147853 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:32.148219 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.149087 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.151013 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.151701 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.151821 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.151966 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:32.152282 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 14:51:32.152406 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432}
I20260620 14:51:32.153558 25160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f9a2c48e66416182d488e2e8460e29" candidate_uuid: "4012070e7f0048a5b20486063abc30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92be6e49c874c1298cade013b065f51" is_pre_election: true
I20260620 14:51:32.153947 25160 raft_consensus.cc:2454] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4012070e7f0048a5b20486063abc30db in term 0.
I20260620 14:51:32.154268 25493 tablet_bootstrap.cc:492] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22: No bootstrap required, opened a new log
I20260620 14:51:32.154521 25493 ts_tablet_manager.cc:1397] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 14:51:32.153419 25435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f9a2c48e66416182d488e2e8460e29" candidate_uuid: "4012070e7f0048a5b20486063abc30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bfab1be9f36478bbf52b29235163b22" is_pre_election: true
I20260620 14:51:32.154736 25234 leader_election.cc:302] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4012070e7f0048a5b20486063abc30db, b92be6e49c874c1298cade013b065f51; no voters:
I20260620 14:51:32.155084 25350 raft_consensus.cc:2781] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:32.155241 25350 raft_consensus.cc:482] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:32.155292 25350 raft_consensus.cc:3037] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [term 0 FOLLOWER]: Advancing to term 1
W20260620 14:51:32.156085 25231 leader_election.cc:341] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049): Illegal state: must be running to vote when last-logged opid is not known
I20260620 14:51:32.156599 25350 raft_consensus.cc:504] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } }
I20260620 14:51:32.157045 25350 leader_election.cc:288] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [CANDIDATE]: Term 1 election: Requested vote from peers 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049), b92be6e49c874c1298cade013b065f51 (127.8.221.193:45499)
I20260620 14:51:32.157444 25435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f9a2c48e66416182d488e2e8460e29" candidate_uuid: "4012070e7f0048a5b20486063abc30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bfab1be9f36478bbf52b29235163b22"
I20260620 14:51:32.157486 25160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f9a2c48e66416182d488e2e8460e29" candidate_uuid: "4012070e7f0048a5b20486063abc30db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92be6e49c874c1298cade013b065f51"
I20260620 14:51:32.157625 25160 raft_consensus.cc:3037] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 0 FOLLOWER]: Advancing to term 1
W20260620 14:51:32.157882 25231 leader_election.cc:341] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049): Illegal state: must be running to vote when last-logged opid is not known
I20260620 14:51:32.158069 25493 raft_consensus.cc:348] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } }
I20260620 14:51:32.158303 25493 raft_consensus.cc:374] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:32.158365 25493 raft_consensus.cc:729] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bfab1be9f36478bbf52b29235163b22, State: Initialized, Role: FOLLOWER
I20260620 14:51:32.158893 25160 raft_consensus.cc:2454] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4012070e7f0048a5b20486063abc30db in term 1.
I20260620 14:51:32.158914 25493 consensus_queue.cc:260] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } }
I20260620 14:51:32.159205 25234 leader_election.cc:302] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4012070e7f0048a5b20486063abc30db, b92be6e49c874c1298cade013b065f51; no voters: 1bfab1be9f36478bbf52b29235163b22
I20260620 14:51:32.159453 25350 raft_consensus.cc:2781] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:32.159706 25350 raft_consensus.cc:686] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [term 1 LEADER]: Becoming Leader. State: Replica: 4012070e7f0048a5b20486063abc30db, State: Running, Role: LEADER
I20260620 14:51:32.159749 25481 heartbeater.cc:503] Master 127.8.221.254:38213 was elected leader, sending a full tablet report...
I20260620 14:51:32.159979 25350 consensus_queue.cc:237] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } }
I20260620 14:51:32.160193 25493 ts_tablet_manager.cc:1428] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:51:32.163321 25017 catalog_manager.cc:5495] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db reported cstate change: term changed from 0 to 1, leader changed from <none> to 4012070e7f0048a5b20486063abc30db (127.8.221.194). New cstate: current_term: 1 leader_uuid: "4012070e7f0048a5b20486063abc30db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } health_report { overall_health: UNKNOWN } } }
I20260620 14:51:32.172911 25496 raft_consensus.cc:482] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:32.173207 25496 raft_consensus.cc:504] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } }
I20260620 14:51:32.174629 25496 leader_election.cc:288] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4012070e7f0048a5b20486063abc30db (127.8.221.194:39433), b92be6e49c874c1298cade013b065f51 (127.8.221.193:45499)
I20260620 14:51:32.179152 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.174825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:39433 (local address 127.8.221.195:49053)
0620 14:51:32.175036 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:32.175053 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:32.175146 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:32.175468 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:32.175471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:32.175487 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:32.175622 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.175628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.176677 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.176680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.177739 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.177748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.178684 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.178688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.178799 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.178824 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:51:32.178903 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:51:32.178991 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":8}
I20260620 14:51:32.179579 25351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.174920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:49053 (local address 127.8.221.194:39433)
0620 14:51:32.175076 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:32.175084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:32.175106 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:51:32.175296 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:32.175300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:32.175365 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:32.175455 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:32.175733 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.176533 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.177910 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.178556 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.178695 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.178839 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:32.179343 (+ 504us) server_negotiation.cc:299] Negotiation successful
0620 14:51:32.179444 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:51:32.179628 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.175391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:32987 (local address 127.8.221.193:45499)
0620 14:51:32.175507 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:32.175512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:32.175913 (+ 401us) server_negotiation.cc:407] Connection header received
0620 14:51:32.176124 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:32.176127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:32.176187 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:32.176287 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:32.176584 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.177152 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.178573 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.179000 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.179131 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.179182 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:32.179393 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:51:32.179481 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:51:32.180075 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.174972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:45499 (local address 127.8.221.195:32987)
0620 14:51:32.175822 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:32.175836 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:32.175933 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:32.176308 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:32.176312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:32.176334 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:32.176474 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.176480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.177279 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.177283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.178460 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.178469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.179124 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.179128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.179244 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.179276 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 14:51:32.179834 (+ 558us) client_negotiation.cc:240] Negotiation successful
0620 14:51:32.179922 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":118,"threads_started":1}
I20260620 14:51:32.180192 25299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f9a2c48e66416182d488e2e8460e29" candidate_uuid: "1bfab1be9f36478bbf52b29235163b22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4012070e7f0048a5b20486063abc30db" is_pre_election: true
I20260620 14:51:32.180483 25160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f9a2c48e66416182d488e2e8460e29" candidate_uuid: "1bfab1be9f36478bbf52b29235163b22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b92be6e49c874c1298cade013b065f51" is_pre_election: true
I20260620 14:51:32.180722 25160 raft_consensus.cc:2379] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1bfab1be9f36478bbf52b29235163b22 in current term 1: Already voted for candidate 4012070e7f0048a5b20486063abc30db in this term.
I20260620 14:51:32.181241 25371 leader_election.cc:302] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1bfab1be9f36478bbf52b29235163b22; no voters: 4012070e7f0048a5b20486063abc30db, b92be6e49c874c1298cade013b065f51
I20260620 14:51:32.181579 25496 raft_consensus.cc:3037] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:32.183336 25496 raft_consensus.cc:2726] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 14:51:32.183933 25500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.178172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54376 (local address 127.8.221.194:39433)
0620 14:51:32.179000 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:32.179008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:32.179021 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:51:32.179105 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:32.179107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:32.179177 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:32.179861 (+ 684us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:32.179962 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.180766 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.181902 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.182246 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.182359 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.182627 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:32.183277 (+ 650us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:32.183418 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:32.183692 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 14:51:32.183800 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":132,"threads_started":1}
I20260620 14:51:32.188319 25160 raft_consensus.cc:1261] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Refusing update from remote peer 4012070e7f0048a5b20486063abc30db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:32.188321 25435 raft_consensus.cc:1261] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22 [term 1 FOLLOWER]: Refusing update from remote peer 4012070e7f0048a5b20486063abc30db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:32.189044 25497 consensus_queue.cc:1035] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:32.189333 25350 consensus_queue.cc:1035] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:32.197966 25501 mvcc.cc:205] Tried to move back new op lower bound from 7298937209594998784 to 7298937209489350656. Current Snapshot: MvccSnapshot[applied={T|T < 7298937209594998784}]
I20260620 14:51:32.207487 25017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:32.207681 25017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:32.210312 25017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c: Sending DeleteTablet for 3 replicas of tablet 35f9a2c48e66416182d488e2e8460e29
I20260620 14:51:32.211032 25279 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f9a2c48e66416182d488e2e8460e29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:32 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:54368
I20260620 14:51:32.211042 25415 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f9a2c48e66416182d488e2e8460e29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:32 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:58166
I20260620 14:51:32.211246 25140 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f9a2c48e66416182d488e2e8460e29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:32 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:59464
I20260620 14:51:32.211583 25352 tablet_replica.cc:330] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db: stopping tablet replica
I20260620 14:51:32.211675 25512 tablet_replica.cc:330] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51: stopping tablet replica
I20260620 14:51:32.211733 25352 raft_consensus.cc:2229] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:51:32.211831 25512 raft_consensus.cc:2229] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:32.211788 25511 tablet_replica.cc:330] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22: stopping tablet replica
I20260620 14:51:32.212055 25352 raft_consensus.cc:2258] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:32.212085 25512 raft_consensus.cc:2258] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:32.212395 25511 raft_consensus.cc:2229] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:32.212677 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 25056
I20260620 14:51:32.213478 25511 raft_consensus.cc:2258] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:32.215339 25511 ts_tablet_manager.cc:1905] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:32.216019 25352 ts_tablet_manager.cc:1905] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:32.218641 25352 ts_tablet_manager.cc:1918] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:32.218726 25352 log.cc:1196] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/35f9a2c48e66416182d488e2e8460e29
I20260620 14:51:32.219128 25352 ts_tablet_manager.cc:1939] T 35f9a2c48e66416182d488e2e8460e29 P 4012070e7f0048a5b20486063abc30db: Deleting consensus metadata
I20260620 14:51:32.219933 25004 catalog_manager.cc:4841] TS 4012070e7f0048a5b20486063abc30db (127.8.221.194:39433): tablet 35f9a2c48e66416182d488e2e8460e29 (table test-table [id=390f00dbf711494d9fbd1aa2ce0d694b]) successfully deleted
I20260620 14:51:32.220206 25511 ts_tablet_manager.cc:1918] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:32.220321 25511 log.cc:1196] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/35f9a2c48e66416182d488e2e8460e29
I20260620 14:51:32.220713 25511 ts_tablet_manager.cc:1939] T 35f9a2c48e66416182d488e2e8460e29 P 1bfab1be9f36478bbf52b29235163b22: Deleting consensus metadata
I20260620 14:51:32.222225 25002 catalog_manager.cc:4841] TS 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049): tablet 35f9a2c48e66416182d488e2e8460e29 (table test-table [id=390f00dbf711494d9fbd1aa2ce0d694b]) successfully deleted
W20260620 14:51:32.222674 25005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.221.193:45499 (error 108)
I20260620 14:51:32.223130 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 25211
I20260620 14:51:32.224676 25213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.224246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:45499 (local address 127.0.0.1:59478)
0620 14:51:32.224463 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:32.224556 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.193:45499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260620 14:51:32.224830 25005 catalog_manager.cc:4584] TS b92be6e49c874c1298cade013b065f51 (127.8.221.193:45499): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35f9a2c48e66416182d488e2e8460e29: Network error: Client connection negotiation failed: client connection to 127.8.221.193:45499: connect: Connection refused (error 111)
I20260620 14:51:32.230602 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 25349
I20260620 14:51:32.237601 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 24985
I20260620 14:51:52.245826 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38213
--webserver_interface=127.8.221.254
--webserver_port=46171
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:38213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:52.375818 25514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:52.376194 25514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:52.376276 25514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:52.380686 25514 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:51:52.380841 25514 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:51:52.380868 25514 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:51:52.380893 25514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:52.380936 25514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:52.380990 25514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:52.381028 25514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:52.381073 25514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:52.387042 25514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:38213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=46171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:51:52.388485 25514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:52.389780 25514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:52.400183 25520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:52.400246 25522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:52.400499 25514 server_base.cc:1034] running on GCE node
W20260620 14:51:52.400193 25519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:52.401321 25514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:52.402706 25514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:52.403926 25514 hybrid_clock.cc:648] HybridClock initialized: now 1781967112403888 us; error 52 us; skew 500 ppm
Jun 20 14:51:52 dist-test-slave-65tz krb5kdc[24438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967112, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:52.407255 25514 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:51:52.408267 25514 webserver.cc:460] Webserver started at http://127.8.221.254:46171/ using document root <none> and password file <none>
I20260620 14:51:52.408928 25514 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:52.408998 25514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:52.413431 25514 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:51:52.415969 25529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:52.417493 25514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:51:52.417654 25514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "cdc7bbdfb08641b794835c384921cd7c"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "4ef63526ad5903b7a719f414cc74974c"
server_key_iv: "5e5d090e9e7b4b56596b9d9cb044b262"
server_key_version: "encryptionkey@0"
I20260620 14:51:52.418352 25514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:52.431252 25514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:52.434561 25514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:52.434767 25514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:52.442719 25514 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:38213
I20260620 14:51:52.443046 25581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:38213 every 8 connection(s)
I20260620 14:51:52.443934 25514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:52.449270 25582 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:51:52.451279 25582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c: Bootstrap starting.
I20260620 14:51:52.454322 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 25514
I20260620 14:51:52.454927 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:45499
--local_ip_for_outbound_sockets=127.8.221.193
--tserver_master_addrs=127.8.221.254:38213
--webserver_port=44527
--webserver_interface=127.8.221.193
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:51:52.472960 25582 log.cc:824] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:52.485455 25582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:51:52.485981 25582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c: Bootstrap complete.
I20260620 14:51:52.490738 25582 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7bbdfb08641b794835c384921cd7c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38213 } }
I20260620 14:51:52.491227 25582 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc7bbdfb08641b794835c384921cd7c, State: Initialized, Role: FOLLOWER
I20260620 14:51:52.491854 25582 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7bbdfb08641b794835c384921cd7c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38213 } }
I20260620 14:51:52.492017 25582 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:52.492110 25582 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:52.492226 25582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:51:52.494609 25582 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7bbdfb08641b794835c384921cd7c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38213 } }
I20260620 14:51:52.494990 25582 leader_election.cc:302] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdc7bbdfb08641b794835c384921cd7c; no voters:
I20260620 14:51:52.495428 25582 leader_election.cc:288] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:51:52.495505 25586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:51:52.495764 25586 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [term 3 LEADER]: Becoming Leader. State: Replica: cdc7bbdfb08641b794835c384921cd7c, State: Running, Role: LEADER
I20260620 14:51:52.496097 25586 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7bbdfb08641b794835c384921cd7c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38213 } }
I20260620 14:51:52.496944 25582 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:52.497331 25587 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdc7bbdfb08641b794835c384921cd7c. Latest consensus state: current_term: 3 leader_uuid: "cdc7bbdfb08641b794835c384921cd7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7bbdfb08641b794835c384921cd7c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38213 } } }
I20260620 14:51:52.497570 25587 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:52.497766 25588 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cdc7bbdfb08641b794835c384921cd7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc7bbdfb08641b794835c384921cd7c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38213 } } }
I20260620 14:51:52.497931 25588 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:52.498270 25590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:52.504570 25590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:52.505268 25590 catalog_manager.cc:1259] Loaded cluster ID: 2ff1698fa3c04f44ad6210cbe10e7a60
I20260620 14:51:52.505334 25590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:52.508227 25590 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:52.509037 25590 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c: Loaded TSK: 0
I20260620 14:51:52.509128 25590 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cdc7bbdfb08641b794835c384921cd7c: Loaded TSK: 1
F20260620 14:51:52.510108 25590 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6529aa09ad google::LogMessage::Fail() at ??:0
@ 0x7f6529aa388a google::LogMessage::SendToLog() at ??:0
@ 0x7f6529aa04de google::LogMessage::Flush() at ??:0
@ 0x7f6529aa1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f652c76418d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f652c75cf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f65344057bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f65343df99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f65343e0178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f6534421df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f653444082b std::function<>::operator()() at ??:0
@ 0x7f65343dfb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f65343e0f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f65343dd579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f6534421640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f652bcf23d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f652bcf2c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f652bcf43f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f652bce3512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f652d3276db start_thread at ??:0
@ 0x7f652859e61f clone at ??:0
W20260620 14:51:52.596709 25584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:52.596971 25584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:52.597013 25584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:52.601475 25584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:52.601524 25584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:52.601586 25584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:51:52.607450 25584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:45499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=44527
--tserver_master_addrs=127.8.221.254:38213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:51:52.608515 25584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:52.609761 25584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:52.620028 25630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:52.620028 25629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:52.620091 25632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:52.620522 25584 server_base.cc:1034] running on GCE node
I20260620 14:51:52.621308 25584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:52.622207 25584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:52.623410 25584 hybrid_clock.cc:648] HybridClock initialized: now 1781967112623385 us; error 45 us; skew 500 ppm
Jun 20 14:51:52 dist-test-slave-65tz krb5kdc[24438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967112, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:52.626653 25584 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:51:52.627751 25584 webserver.cc:460] Webserver started at http://127.8.221.193:44527/ using document root <none> and password file <none>
I20260620 14:51:52.628449 25584 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:52.628545 25584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:52.633515 25584 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:51:52.636523 25639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:52.638283 25584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:51:52.638432 25584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "b92be6e49c874c1298cade013b065f51"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "3439e63d12178a49e12b492223efeeef"
server_key_iv: "c625ce12042ccb186525bbc77542df8d"
server_key_version: "encryptionkey@0"
I20260620 14:51:52.639098 25584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:52.665357 25584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:52.669166 25584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:52.669400 25584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:52.670131 25584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:52.671469 25646 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:51:52.677368 25584 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:51:52.677471 25584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 14:51:52.677562 25584 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:51:52.679764 25584 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:51:52.679836 25584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 14:51:52.680230 25646 tablet_bootstrap.cc:492] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51: Bootstrap starting.
I20260620 14:51:52.692377 25584 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:45499
I20260620 14:51:52.692683 25753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:45499 every 8 connection(s)
I20260620 14:51:52.693689 25584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:51:52.703773 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 25584
I20260620 14:51:52.704636 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:39433
--local_ip_for_outbound_sockets=127.8.221.194
--tserver_master_addrs=127.8.221.254:38213
--webserver_port=37501
--webserver_interface=127.8.221.194
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:52.785395 25756 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:52.696110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:33089)
0620 14:51:52.696660 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:52.696717 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:52.697685 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:52.783936 (+ 86251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":156,"threads_started":1}
W20260620 14:51:52.786639 25754 heartbeater.cc:644] Failed to heartbeat to 127.8.221.254:38213 (0 consecutive failures): Network error: Failed to ping master at 127.8.221.254:38213: Client connection negotiation failed: client connection to 127.8.221.254:38213: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:51:52.788671 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:52.788266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:51219)
0620 14:51:52.788455 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:52.788555 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 14:51:52.789487 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:52.789246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:37117)
0620 14:51:52.789379 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:52.789429 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 14:51:52.789731 25754 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:51:52.836455 25757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:52.836802 25757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:52.836889 25757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:52.841243 25757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:52.841331 25757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:52.841425 25757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:51:52.847199 25757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:39433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=37501
--tserver_master_addrs=127.8.221.254:38213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:51:52.848639 25757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:52.850009 25757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:52.861593 25762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:52.861984 25757 server_base.cc:1034] running on GCE node
W20260620 14:51:52.861593 25763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:52.861706 25765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:52.862597 25757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:52.863430 25757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:52.864615 25757 hybrid_clock.cc:648] HybridClock initialized: now 1781967112864583 us; error 46 us; skew 500 ppm
Jun 20 14:51:52 dist-test-slave-65tz krb5kdc[24438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967112, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:52.867802 25757 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:51:52.869021 25757 webserver.cc:460] Webserver started at http://127.8.221.194:37501/ using document root <none> and password file <none>
I20260620 14:51:52.869745 25757 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:52.869880 25757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:52.874691 25757 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 14:51:52.877199 25772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:52.878829 25757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:51:52.878937 25757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "4012070e7f0048a5b20486063abc30db"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "8ca0b741c9b79cf9721b330927db9d87"
server_key_iv: "a0de6a1873627d1bd21f05e28ef66043"
server_key_version: "encryptionkey@0"
I20260620 14:51:52.879644 25757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:52.886826 25646 log_util.cc:230] Ignoring log segment corruption in /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal/wals/35f9a2c48e66416182d488e2e8460e29.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 577 for log segment /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-0/wal/wals/35f9a2c48e66416182d488e2e8460e29.recovery/wal-000000001: Prior entries: [off=506 REPLICATE (1.1)] [off=506 REPLICATE (1.2)] [off=537 COMMIT (1.1)] [off=577 COMMIT (1.2)]
I20260620 14:51:52.888876 25646 log.cc:824] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:52.896622 25646 tablet_bootstrap.cc:492] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:51:52.896989 25646 tablet_bootstrap.cc:492] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51: Bootstrap complete.
I20260620 14:51:52.898185 25646 ts_tablet_manager.cc:1397] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51: Time spent bootstrapping tablet: real 0.218s user 0.205s sys 0.012s
I20260620 14:51:52.900509 25646 raft_consensus.cc:348] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } }
I20260620 14:51:52.900935 25646 raft_consensus.cc:729] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92be6e49c874c1298cade013b065f51, State: Initialized, Role: FOLLOWER
I20260620 14:51:52.901422 25646 consensus_queue.cc:260] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } }
I20260620 14:51:52.902386 25646 ts_tablet_manager.cc:1428] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 14:51:52.902978 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:52.902697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:53169)
0620 14:51:52.902839 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:52.902908 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 14:51:52.903693 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:52.903487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:51653)
0620 14:51:52.903588 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:52.903634 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 14:51:52.912846 25757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:52.916657 25757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:52.916903 25757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:52.917683 25757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:52.918736 25757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:52.918812 25757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:52.918889 25757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:52.918941 25757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:52.930852 25757 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:39433
I20260620 14:51:52.930920 25887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:39433 every 8 connection(s)
I20260620 14:51:52.932185 25757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:51:52.936612 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:52.934631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:33223)
0620 14:51:52.935256 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:52.935747 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":180,"threads_started":1}
W20260620 14:51:52.937139 25888 heartbeater.cc:644] Failed to heartbeat to 127.8.221.254:38213 (0 consecutive failures): Network error: Failed to ping master at 127.8.221.254:38213: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
I20260620 14:51:52.937893 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:52.937568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:57727)
0620 14:51:52.937690 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:52.937816 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 14:51:52.938099 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 25757
I20260620 14:51:52.938669 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:52.938422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:59909)
0620 14:51:52.938543 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:52.938604 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 14:51:52.938663 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:40049
--local_ip_for_outbound_sockets=127.8.221.195
--tserver_master_addrs=127.8.221.254:38213
--webserver_port=44845
--webserver_interface=127.8.221.195
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:52.938913 25888 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:51:53.067421 25891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:53.067741 25891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:53.067790 25891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:53.072053 25891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:53.072129 25891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:53.072194 25891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:51:53.077819 25891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:40049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=44845
--tserver_master_addrs=127.8.221.254:38213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:51:53.079176 25891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:53.080454 25891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:53.092133 25896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:53.092164 25899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:53.092164 25897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:53.092720 25891 server_base.cc:1034] running on GCE node
I20260620 14:51:53.093212 25891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:53.094030 25891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:53.095216 25891 hybrid_clock.cc:648] HybridClock initialized: now 1781967113095190 us; error 44 us; skew 500 ppm
Jun 20 14:51:53 dist-test-slave-65tz krb5kdc[24438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967113, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:53.098299 25891 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:51:53.099248 25891 webserver.cc:460] Webserver started at http://127.8.221.195:44845/ using document root <none> and password file <none>
I20260620 14:51:53.099887 25891 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:53.099953 25891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:53.104188 25891 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:51:53.106619 25906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:53.108157 25891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:51:53.108280 25891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "1bfab1be9f36478bbf52b29235163b22"
format_stamp: "Formatted at 2026-06-20 14:51:10 on dist-test-slave-65tz"
server_key: "fb59d68d07fa8c5e512c568485f77a9a"
server_key_iv: "a0eea83d54321860fa484a4eeb9c542b"
server_key_version: "encryptionkey@0"
I20260620 14:51:53.109016 25891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:53.128171 25891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:53.131726 25891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:53.131906 25891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:53.132483 25891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:53.133391 25891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:53.133443 25891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:53.133515 25891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:53.133553 25891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:53.144479 25891 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:40049
I20260620 14:51:53.144579 26019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:40049 every 8 connection(s)
I20260620 14:51:53.145654 25891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:51:53.147012 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 25891
I20260620 14:51:53.150554 26023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:53.148183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.195:47983)
0620 14:51:53.148838 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:53.149411 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":161,"threads_started":1}
W20260620 14:51:53.151386 26020 heartbeater.cc:644] Failed to heartbeat to 127.8.221.254:38213 (0 consecutive failures): Network error: Failed to ping master at 127.8.221.254:38213: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
I20260620 14:51:53.152297 26023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:53.151994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.195:34995)
0620 14:51:53.152157 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:53.152219 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 14:51:53.153227 26023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:53.152987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.195:53079)
0620 14:51:53.153123 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:53.153172 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 14:51:53.153532 26020 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:51:53.905083 26024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:53.904440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:58669)
0620 14:51:53.904830 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:53.904932 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":91,"threads_started":1}
I20260620 14:51:53.940214 26025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:53.939587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:53767)
0620 14:51:53.939917 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:53.940017 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":106,"threads_started":1}
I20260620 14:51:54.154711 26026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.154187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.195:42965)
0620 14:51:54.154526 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:54.154591 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":154,"threads_started":1}
I20260620 14:51:54.694525 26027 raft_consensus.cc:482] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:54.694784 26027 raft_consensus.cc:504] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } }
I20260620 14:51:54.695943 26027 leader_election.cc:288] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049), 4012070e7f0048a5b20486063abc30db (127.8.221.194:39433)
Jun 20 14:51:54 dist-test-slave-65tz krb5kdc[24438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967112, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.194@KRBTEST.COM
Jun 20 14:51:54 dist-test-slave-65tz krb5kdc[24438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967112, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.195@KRBTEST.COM
I20260620 14:51:54.718598 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.696566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:55733 (local address 127.8.221.194:39433)
0620 14:51:54.696988 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:54.697004 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:54.697056 (+ 52us) server_negotiation.cc:407] Connection header received
0620 14:51:54.697953 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:54.697960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:54.698343 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:54.698605 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:54.700363 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.701995 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.704310 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.705083 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.705262 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.712850 (+ 7588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:54.712857 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:54.712882 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:54.712931 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:54.715235 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:54.715980 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:54.715987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:54.715994 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:54.716151 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:54.716608 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:54.716611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:54.716612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:54.717138 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:54.717258 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:54.718070 (+ 812us) server_negotiation.cc:299] Negotiation successful
0620 14:51:54.718372 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":80,"threads_started":1}
I20260620 14:51:54.718623 26029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.696388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:39433 (local address 127.8.221.193:55733)
0620 14:51:54.696838 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:54.696859 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:54.697013 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:54.698761 (+ 1748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:54.698769 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:54.699494 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:54.700104 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.700134 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.702191 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.702195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.704157 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.704166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.705248 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.705252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.705546 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.709665 (+ 4119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:54.709707 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:54.712507 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:54.715393 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:54.715405 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:54.715425 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:54.715852 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:54.716279 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:54.716283 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:54.716285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:54.716493 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:54.717289 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:54.717297 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:54.717610 (+ 313us) client_negotiation.cc:769] Sending connection context
0620 14:51:54.717931 (+ 321us) client_negotiation.cc:240] Negotiation successful
0620 14:51:54.718389 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":284,"spinlock_wait_cycles":26112,"thread_start_us":114,"threads_started":1}
I20260620 14:51:54.718637 26031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.696566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:35709 (local address 127.8.221.195:40049)
0620 14:51:54.697023 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:54.697047 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:54.697205 (+ 158us) server_negotiation.cc:407] Connection header received
0620 14:51:54.698278 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:54.698286 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:54.698767 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:54.699156 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:54.700359 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.701999 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.704310 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.705083 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.705262 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.713314 (+ 8052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:54.713320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:54.713335 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:54.713408 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:54.715616 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:54.716252 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:54.716262 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:54.716268 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:54.716471 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:54.716863 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:54.716866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:54.716867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:54.717449 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:54.717652 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:54.718064 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 14:51:54.718379 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":126,"threads_started":1}
I20260620 14:51:54.718725 26028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.696388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:40049 (local address 127.8.221.193:35709)
0620 14:51:54.697025 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:54.697043 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:54.697220 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:54.699127 (+ 1907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:54.699130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:54.699472 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:54.700115 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.700132 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.702191 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.702195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.704153 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.704166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.705248 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.705252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.705550 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.709665 (+ 4115us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:54.709701 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:54.713064 (+ 3363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:54.715776 (+ 2712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:54.715779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:54.715781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:54.716088 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:54.716621 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:54.716624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:54.716626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:54.716754 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:54.717637 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:54.717639 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:54.717758 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 14:51:54.717931 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 14:51:54.718498 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":599,"spinlock_wait_cycles":22400,"thread_start_us":85,"threads_started":1}
W20260620 14:51:54.722057 25642 leader_election.cc:341] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4012070e7f0048a5b20486063abc30db (127.8.221.194:39433): Not found: Tablet not found: 35f9a2c48e66416182d488e2e8460e29
W20260620 14:51:54.722255 25640 leader_election.cc:341] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049): Not found: Tablet not found: 35f9a2c48e66416182d488e2e8460e29
I20260620 14:51:54.722350 25640 leader_election.cc:302] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b92be6e49c874c1298cade013b065f51; no voters: 1bfab1be9f36478bbf52b29235163b22, 4012070e7f0048a5b20486063abc30db
I20260620 14:51:54.722679 26027 raft_consensus.cc:2726] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:51:54.906605 26029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.906221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:55575)
0620 14:51:54.906398 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:54.906481 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 14:51:54.942009 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.941344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:55547)
0620 14:51:54.941758 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:54.941897 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":153,"threads_started":1}
I20260620 14:51:55.156392 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:55.155793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.195:48575)
0620 14:51:55.156173 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:55.156259 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":129,"threads_started":1}
I20260620 14:51:55.908390 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:55.907623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:41287)
0620 14:51:55.908218 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:55.908282 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":133,"threads_started":1}
I20260620 14:51:55.943568 26035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:55.943015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:33245)
0620 14:51:55.943337 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:55.943435 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":82,"threads_started":1}
I20260620 14:51:56.158054 26036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:56.157364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.195:45569)
0620 14:51:56.157747 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:56.157915 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":97,"threads_started":1}
I20260620 14:51:56.474583 26037 raft_consensus.cc:482] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:56.474763 26037 raft_consensus.cc:504] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } }
I20260620 14:51:56.475387 26037 leader_election.cc:288] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049), 4012070e7f0048a5b20486063abc30db (127.8.221.194:39433)
W20260620 14:51:56.476622 25640 leader_election.cc:341] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049): Not found: Tablet not found: 35f9a2c48e66416182d488e2e8460e29
W20260620 14:51:56.476821 25642 leader_election.cc:341] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4012070e7f0048a5b20486063abc30db (127.8.221.194:39433): Not found: Tablet not found: 35f9a2c48e66416182d488e2e8460e29
I20260620 14:51:56.476938 25642 leader_election.cc:302] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b92be6e49c874c1298cade013b065f51; no voters: 1bfab1be9f36478bbf52b29235163b22, 4012070e7f0048a5b20486063abc30db
I20260620 14:51:56.477191 26037 raft_consensus.cc:2726] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:51:56.909987 26038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:56.909449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:44643)
0620 14:51:56.909728 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:56.909790 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
I20260620 14:51:56.945000 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:56.944421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:42001)
0620 14:51:56.944811 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:56.944881 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":139,"threads_started":1}
I20260620 14:51:57.159531 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:57.158987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.195:58975)
0620 14:51:57.159315 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:57.159393 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":84,"threads_started":1}
I20260620 14:51:57.911756 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:57.911200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:36191)
0620 14:51:57.911553 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:57.911633 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":153,"threads_started":1}
I20260620 14:51:57.946336 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:57.945909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:38347)
0620 14:51:57.946170 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:57.946228 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":100,"threads_started":1}
I20260620 14:51:58.161402 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:58.160681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.195:34115)
0620 14:51:58.161129 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:58.161239 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":111,"threads_started":1}
I20260620 14:51:58.913524 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:58.912921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:45595)
0620 14:51:58.913278 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:58.913389 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
I20260620 14:51:58.937529 26045 raft_consensus.cc:482] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:58.937690 26045 raft_consensus.cc:504] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } }
I20260620 14:51:58.938288 26045 leader_election.cc:288] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049), 4012070e7f0048a5b20486063abc30db (127.8.221.194:39433)
W20260620 14:51:58.939280 25640 leader_election.cc:341] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049): Not found: Tablet not found: 35f9a2c48e66416182d488e2e8460e29
W20260620 14:51:58.939456 25642 leader_election.cc:341] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4012070e7f0048a5b20486063abc30db (127.8.221.194:39433): Not found: Tablet not found: 35f9a2c48e66416182d488e2e8460e29
I20260620 14:51:58.939528 25642 leader_election.cc:302] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b92be6e49c874c1298cade013b065f51; no voters: 1bfab1be9f36478bbf52b29235163b22, 4012070e7f0048a5b20486063abc30db
I20260620 14:51:58.939679 26045 raft_consensus.cc:2726] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:51:58.947691 26046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:58.947262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:40727)
0620 14:51:58.947540 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:58.947603 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":69,"threads_started":1}
I20260620 14:51:59.163036 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:59.162375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.195:51533)
0620 14:51:59.162791 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:59.162892 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":139,"threads_started":1}
I20260620 14:51:59.915278 26048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:59.914746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:49995)
0620 14:51:59.915094 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:59.915166 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":101,"threads_started":1}
I20260620 14:51:59.949234 26049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:59.948622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:49889)
0620 14:51:59.949009 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:59.949123 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":101,"threads_started":1}
I20260620 14:52:00.164804 26050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:00.164189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.195:49639)
0620 14:52:00.164578 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:00.164675 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":91,"threads_started":1}
I20260620 14:52:00.593297 26051 raft_consensus.cc:482] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:52:00.593519 26051 raft_consensus.cc:504] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } }
I20260620 14:52:00.594074 26051 leader_election.cc:288] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049), 4012070e7f0048a5b20486063abc30db (127.8.221.194:39433)
W20260620 14:52:00.595220 25642 leader_election.cc:341] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4012070e7f0048a5b20486063abc30db (127.8.221.194:39433): Not found: Tablet not found: 35f9a2c48e66416182d488e2e8460e29
W20260620 14:52:00.595403 25640 leader_election.cc:341] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049): Not found: Tablet not found: 35f9a2c48e66416182d488e2e8460e29
I20260620 14:52:00.595487 25640 leader_election.cc:302] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b92be6e49c874c1298cade013b065f51; no voters: 1bfab1be9f36478bbf52b29235163b22, 4012070e7f0048a5b20486063abc30db
I20260620 14:52:00.595652 26051 raft_consensus.cc:2726] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:52:00.916968 26052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:00.916390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:51321)
0620 14:52:00.916749 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:00.916862 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":122,"threads_started":1}
I20260620 14:52:00.950766 26053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:00.950249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:41473)
0620 14:52:00.950586 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:00.950661 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":104,"threads_started":1}
I20260620 14:52:01.166431 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:01.165790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.195:38939)
0620 14:52:01.166222 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:01.166313 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":94,"threads_started":1}
I20260620 14:52:01.918481 26055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:01.917950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:36725)
0620 14:52:01.918300 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:01.918377 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":116,"threads_started":1}
I20260620 14:52:01.952304 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:01.951740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:47987)
0620 14:52:01.952112 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:01.952180 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":159,"threads_started":1}
I20260620 14:52:02.168259 26057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:02.167562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.195:46365)
0620 14:52:02.167965 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:02.168118 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":147,"threads_started":1}
I20260620 14:52:02.268149 26058 raft_consensus.cc:482] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:52:02.268316 26058 raft_consensus.cc:504] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfab1be9f36478bbf52b29235163b22" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 40049 } } peers { permanent_uuid: "4012070e7f0048a5b20486063abc30db" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 39433 } } peers { permanent_uuid: "b92be6e49c874c1298cade013b065f51" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45499 } }
I20260620 14:52:02.268879 26058 leader_election.cc:288] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049), 4012070e7f0048a5b20486063abc30db (127.8.221.194:39433)
W20260620 14:52:02.270119 25640 leader_election.cc:341] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1bfab1be9f36478bbf52b29235163b22 (127.8.221.195:40049): Not found: Tablet not found: 35f9a2c48e66416182d488e2e8460e29
W20260620 14:52:02.270309 25642 leader_election.cc:341] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4012070e7f0048a5b20486063abc30db (127.8.221.194:39433): Not found: Tablet not found: 35f9a2c48e66416182d488e2e8460e29
I20260620 14:52:02.270361 25640 leader_election.cc:302] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b92be6e49c874c1298cade013b065f51; no voters: 1bfab1be9f36478bbf52b29235163b22, 4012070e7f0048a5b20486063abc30db
I20260620 14:52:02.270561 26058 raft_consensus.cc:2726] T 35f9a2c48e66416182d488e2e8460e29 P b92be6e49c874c1298cade013b065f51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:52:02.920220 26059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:02.919661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:60371)
0620 14:52:02.919998 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:02.920087 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
I20260620 14:52:02.954017 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:02.953389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:48935)
0620 14:52:02.953763 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:02.953906 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":102,"threads_started":1}
I20260620 14:52:03.170459 26061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:03.169518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.195:34927)
0620 14:52:03.170166 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:03.170254 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":150,"threads_started":1}
I20260620 14:52:03.921729 26062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:03.921139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:54803)
0620 14:52:03.921529 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:03.921616 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":121,"threads_started":1}
I20260620 14:52:03.955497 26063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:03.954974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:51027)
0620 14:52:03.955288 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:03.955367 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":121,"threads_started":1}
I20260620 14:52:04.172269 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:04.171795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.195:51359)
0620 14:52:04.172084 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:04.172155 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":103,"threads_started":1}
I20260620 14:52:04.923498 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:04.922859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:45703)
0620 14:52:04.923277 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:04.923365 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":92,"threads_started":1}
I20260620 14:52:04.957226 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:04.956644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:41655)
0620 14:52:04.957024 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:04.957112 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":105,"threads_started":1}
I20260620 14:52:05.173676 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:05.173167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.195:32785)
0620 14:52:05.173475 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:05.173552 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
I20260620 14:52:05.925202 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:05.924590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:38541)
0620 14:52:05.924990 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:05.925074 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":111,"threads_started":1}
I20260620 14:52:05.958825 26069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:05.958262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:45437)
0620 14:52:05.958619 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:05.958692 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":126,"threads_started":1}
I20260620 14:52:06.175246 26070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:06.174685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.195:44717)
0620 14:52:06.175057 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:06.175121 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":120,"threads_started":1}
I20260620 14:52:06.926697 26071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:06.926161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:50273)
0620 14:52:06.926492 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:06.926561 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":83,"threads_started":1}
I20260620 14:52:06.960108 26072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:06.959679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:58021)
0620 14:52:06.959956 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:06.960012 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
I20260620 14:52:07.177008 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:07.176412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.195:59279)
0620 14:52:07.176781 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:07.176873 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":87,"threads_started":1}
I20260620 14:52:07.928439 26074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:07.927924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.193:54205)
0620 14:52:07.928231 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:07.928313 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":78,"threads_started":1}
I20260620 14:52:07.962069 26075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:07.961344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38213 (local address 127.8.221.194:52603)
0620 14:52:07.961712 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:07.961888 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.254:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":88,"threads_started":1}
I20260620 14:52:08.153373 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 25584
I20260620 14:52:08.160017 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 25757
I20260620 14:52:08.165962 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 25891
2026-06-20T14:52:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59542 ms)
[----------] 30 tests from SecurityITest (256701 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:08 dist-test-slave-65tz krb5kdc[26078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:08 dist-test-slave-65tz krb5kdc[26078](info): set up 2 sockets
Jun 20 14:52:08 dist-test-slave-65tz krb5kdc[26078](info): commencing operation
krb5kdc: starting...
W20260620 14:52:09.250059 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:09 dist-test-slave-65tz krb5kdc[26078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:52:09.388355 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:35427
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:33501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:35427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:09.524519 26091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:09.524873 26091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:09.524920 26091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:09.529318 26091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:09.529430 26091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:09.529451 26091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:09.529470 26091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:09.529493 26091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:09.535763 26091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:33501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:35427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:35427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:09.537286 26091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:09.538568 26091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:09.549711 26099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:09.549788 26097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:09.550000 26091 server_base.cc:1034] running on GCE node
W20260620 14:52:09.549810 26096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:09.550887 26091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:09.552189 26091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:09.553395 26091 hybrid_clock.cc:648] HybridClock initialized: now 1781967129553372 us; error 45 us; skew 500 ppm
Jun 20 14:52:09 dist-test-slave-65tz krb5kdc[26078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967129, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:09.556577 26091 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:52:09.557593 26091 webserver.cc:460] Webserver started at http://127.8.221.254:40795/ using document root <none> and password file <none>
I20260620 14:52:09.558238 26091 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:09.558306 26091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:09.558480 26091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:09.560591 26091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "0a578fc007e349dc9b209955f0206752"
format_stamp: "Formatted at 2026-06-20 14:52:09 on dist-test-slave-65tz"
server_key: "ee9c7a1e3b262916202e976e1ed25de1"
server_key_iv: "8ddb2cb8a192cd686b667c5de1c83c97"
server_key_version: "encryptionkey@0"
I20260620 14:52:09.561123 26091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "0a578fc007e349dc9b209955f0206752"
format_stamp: "Formatted at 2026-06-20 14:52:09 on dist-test-slave-65tz"
server_key: "ee9c7a1e3b262916202e976e1ed25de1"
server_key_iv: "8ddb2cb8a192cd686b667c5de1c83c97"
server_key_version: "encryptionkey@0"
I20260620 14:52:09.565389 26091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:52:09.568327 26106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:09.569927 26091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:09.570086 26091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "0a578fc007e349dc9b209955f0206752"
format_stamp: "Formatted at 2026-06-20 14:52:09 on dist-test-slave-65tz"
server_key: "ee9c7a1e3b262916202e976e1ed25de1"
server_key_iv: "8ddb2cb8a192cd686b667c5de1c83c97"
server_key_version: "encryptionkey@0"
I20260620 14:52:09.570209 26091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:09.591495 26091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:09.595007 26091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:09.595225 26091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:09.603086 26091 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:35427
I20260620 14:52:09.603116 26158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:35427 every 8 connection(s)
I20260620 14:52:09.604086 26091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:09.607620 26159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:09.607875 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 26091
I20260620 14:52:09.608031 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:09.608397 9079 external_mini_cluster.cc:1428] Setting key c4b65034110c033c0a04bd4434f877cb
I20260620 14:52:09.615568 26159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752: Bootstrap starting.
Jun 20 14:52:09 dist-test-slave-65tz krb5kdc[26078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:52:09.618458 26159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:09.619447 26159 log.cc:824] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:09.622327 26159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752: No bootstrap required, opened a new log
I20260620 14:52:09.624706 26162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:09.610181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53642 (local address 127.8.221.254:35427)
0620 14:52:09.610794 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:09.610811 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:09.610852 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:52:09.611759 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:09.611788 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:09.612166 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:09.612536 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:09.613384 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:09.614491 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:09.615375 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:09.615932 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:09.616108 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:09.619136 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:09.619141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:09.619157 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:09.619185 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:09.621145 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:09.621911 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:09.621916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:09.621922 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:09.622051 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:09.622492 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:09.622499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:09.622501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:09.622991 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:09.623167 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:09.623527 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 14:52:09.623874 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":186,"threads_started":1}
I20260620 14:52:09.627040 26159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a578fc007e349dc9b209955f0206752" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 35427 } }
I20260620 14:52:09.627319 26159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:09.627415 26159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a578fc007e349dc9b209955f0206752, State: Initialized, Role: FOLLOWER
I20260620 14:52:09.627925 26159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a578fc007e349dc9b209955f0206752" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 35427 } }
I20260620 14:52:09.628063 26159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:09.628136 26159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:09.628248 26159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:09.629382 26159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a578fc007e349dc9b209955f0206752" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 35427 } }
I20260620 14:52:09.629880 26159 leader_election.cc:302] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a578fc007e349dc9b209955f0206752; no voters:
I20260620 14:52:09.630369 26159 leader_election.cc:288] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:09.630537 26164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:09.630843 26164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752 [term 1 LEADER]: Becoming Leader. State: Replica: 0a578fc007e349dc9b209955f0206752, State: Running, Role: LEADER
I20260620 14:52:09.631273 26164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a578fc007e349dc9b209955f0206752" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 35427 } }
I20260620 14:52:09.631982 26159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:09.633345 26165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a578fc007e349dc9b209955f0206752" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a578fc007e349dc9b209955f0206752" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 35427 } } }
I20260620 14:52:09.633498 26165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:09.634080 26166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a578fc007e349dc9b209955f0206752. Latest consensus state: current_term: 1 leader_uuid: "0a578fc007e349dc9b209955f0206752" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a578fc007e349dc9b209955f0206752" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 35427 } } }
I20260620 14:52:09.634187 26166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:09.634375 26173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:09.637689 26173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:09.643664 26173 catalog_manager.cc:1347] Generated new cluster ID: 1de26798962d4778b857521f660719d9
I20260620 14:52:09.643743 26173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:09.652467 26173 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:09.653600 26173 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:09.665436 26173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752: Generated new TSK 0
I20260620 14:52:09.666579 26173 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
I20260620 14:52:09.708657 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:35427
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:33501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:09.841017 26185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:09.841353 26185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:09.841403 26185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:09.845582 26185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:09.845645 26185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:09.845708 26185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:52:09.851747 26185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:33501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:35427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:09.853024 26185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:09.854286 26185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:09.866693 26191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:09.866621 26193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:09.866706 26190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:09.866840 26185 server_base.cc:1034] running on GCE node
I20260620 14:52:09.867910 26185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:09.868682 26185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:09.869904 26185 hybrid_clock.cc:648] HybridClock initialized: now 1781967129869869 us; error 50 us; skew 500 ppm
Jun 20 14:52:09 dist-test-slave-65tz krb5kdc[26078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967129, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:09.873251 26185 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:52:09.874519 26185 webserver.cc:460] Webserver started at http://127.8.221.193:44841/ using document root <none> and password file <none>
I20260620 14:52:09.875221 26185 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:09.875327 26185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:09.875576 26185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:09.877957 26185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "6cb4416263e547b2add25682544cbd19"
format_stamp: "Formatted at 2026-06-20 14:52:09 on dist-test-slave-65tz"
server_key: "746e4ff3697273456bc97def1035f007"
server_key_iv: "e090f6bd12ac6aa2fc9d445b0df38dab"
server_key_version: "encryptionkey@0"
I20260620 14:52:09.878595 26185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "6cb4416263e547b2add25682544cbd19"
format_stamp: "Formatted at 2026-06-20 14:52:09 on dist-test-slave-65tz"
server_key: "746e4ff3697273456bc97def1035f007"
server_key_iv: "e090f6bd12ac6aa2fc9d445b0df38dab"
server_key_version: "encryptionkey@0"
I20260620 14:52:09.883219 26185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:52:09.886076 26200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:09.887580 26185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:52:09.887733 26185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "6cb4416263e547b2add25682544cbd19"
format_stamp: "Formatted at 2026-06-20 14:52:09 on dist-test-slave-65tz"
server_key: "746e4ff3697273456bc97def1035f007"
server_key_iv: "e090f6bd12ac6aa2fc9d445b0df38dab"
server_key_version: "encryptionkey@0"
I20260620 14:52:09.887852 26185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:09.912500 26185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:09.915853 26185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:09.916070 26185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:09.916759 26185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:09.917658 26185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:09.917735 26185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:09.917806 26185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:09.917892 26185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:09.927953 26185 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:45039
I20260620 14:52:09.927989 26313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:45039 every 8 connection(s)
I20260620 14:52:09.928920 26185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:52:09.937932 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 26185
I20260620 14:52:09.938130 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:52:09.938519 9079 external_mini_cluster.cc:1428] Setting key 5e4465d94358596f41e357c53a1fda2d
Jun 20 14:52:09 dist-test-slave-65tz krb5kdc[26078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967129, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:52:09.947268 26162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:09.931274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:59677 (local address 127.8.221.254:35427)
0620 14:52:09.931448 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:09.931469 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:09.932423 (+ 954us) server_negotiation.cc:407] Connection header received
0620 14:52:09.934271 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:09.934274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:09.934335 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:09.934428 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:09.935955 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:09.936530 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:09.938094 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:09.938435 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:09.938526 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:09.941783 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:09.941787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:09.941790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:09.941849 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:09.943852 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:09.944843 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:09.944845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:09.944847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:09.944935 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:09.945467 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:09.945473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:09.945477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:09.945732 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:09.945883 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:09.946830 (+ 947us) server_negotiation.cc:299] Negotiation successful
0620 14:52:09.947013 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:52:09.948376 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:09.931683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:35427 (local address 127.8.221.193:59677)
0620 14:52:09.932209 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:09.932265 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:09.933986 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:09.934643 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:09.934671 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:09.935174 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:09.935789 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:09.935805 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:09.936666 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:09.936669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:09.937917 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:09.937930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:09.938624 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:09.938629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:09.938844 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:09.939631 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:09.939651 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:09.941588 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:09.944054 (+ 2466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:09.944067 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:09.944097 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:09.944672 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:09.945051 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:09.945058 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:09.945063 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:09.945307 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:09.945917 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:09.945929 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:09.946467 (+ 538us) client_negotiation.cc:769] Sending connection context
0620 14:52:09.946762 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 14:52:09.947223 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":106,"threads_started":1}
I20260620 14:52:09.950251 26314 heartbeater.cc:347] Connected to a master server at 127.8.221.254:35427
I20260620 14:52:09.950662 26314 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.221.194@KRBTEST.COM; defaulting to no policy
I20260620 14:52:09.951458 26314 heartbeater.cc:511] Master 127.8.221.254:35427 requested a full tablet report, sending...
I20260620 14:52:09.953404 26123 ts_manager.cc:194] Registered new tserver with Master: 6cb4416263e547b2add25682544cbd19 (127.8.221.193:45039)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.194@KRBTEST.COM" created.
I20260620 14:52:09.955075 26123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:59677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
I20260620 14:52:09.970741 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:35427
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:33501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:10.089427 26319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:10.089700 26319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:10.089742 26319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:10.093983 26319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:10.094051 26319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:10.094112 26319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:52:10.099471 26319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:33501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:35427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:10.100553 26319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:10.101615 26319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:10.112614 26327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:10.112607 26325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:10.112685 26319 server_base.cc:1034] running on GCE node
W20260620 14:52:10.112607 26324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:10.113356 26319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:10.114085 26319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:10.115279 26319 hybrid_clock.cc:648] HybridClock initialized: now 1781967130115254 us; error 40 us; skew 500 ppm
Jun 20 14:52:10 dist-test-slave-65tz krb5kdc[26078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967130, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:10.118254 26319 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:52:10.119204 26319 webserver.cc:460] Webserver started at http://127.8.221.194:37251/ using document root <none> and password file <none>
I20260620 14:52:10.119741 26319 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:10.119807 26319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:10.119976 26319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:10.122192 26319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "4b857148ce3345e8bb6122fb68185290"
format_stamp: "Formatted at 2026-06-20 14:52:10 on dist-test-slave-65tz"
server_key: "c2207c728abd99bf2335156cc81c0858"
server_key_iv: "0e8f3d5d1a6eeeeb942e87ae85ae4ebc"
server_key_version: "encryptionkey@0"
I20260620 14:52:10.122736 26319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "4b857148ce3345e8bb6122fb68185290"
format_stamp: "Formatted at 2026-06-20 14:52:10 on dist-test-slave-65tz"
server_key: "c2207c728abd99bf2335156cc81c0858"
server_key_iv: "0e8f3d5d1a6eeeeb942e87ae85ae4ebc"
server_key_version: "encryptionkey@0"
I20260620 14:52:10.126762 26319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:52:10.129386 26334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:10.130847 26319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:52:10.130981 26319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "4b857148ce3345e8bb6122fb68185290"
format_stamp: "Formatted at 2026-06-20 14:52:10 on dist-test-slave-65tz"
server_key: "c2207c728abd99bf2335156cc81c0858"
server_key_iv: "0e8f3d5d1a6eeeeb942e87ae85ae4ebc"
server_key_version: "encryptionkey@0"
I20260620 14:52:10.131073 26319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:10.143282 26319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:10.146425 26319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:10.146597 26319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:10.147173 26319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:10.148072 26319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:10.148125 26319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:10.148171 26319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:10.148198 26319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:10.159392 26319 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:41417
I20260620 14:52:10.159412 26447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:41417 every 8 connection(s)
I20260620 14:52:10.160408 26319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:52:10.168545 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 26319
I20260620 14:52:10.168715 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:52:10.169073 9079 external_mini_cluster.cc:1428] Setting key e80a5658a097b395091f3f46e2362272
Jun 20 14:52:10 dist-test-slave-65tz krb5kdc[26078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967130, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:52:10.178094 26162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.162837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:57169 (local address 127.8.221.254:35427)
0620 14:52:10.163047 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:10.163052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:10.164018 (+ 966us) server_negotiation.cc:407] Connection header received
0620 14:52:10.165643 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:10.165655 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:10.165739 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:10.165921 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:10.167513 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.168112 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.169931 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.170155 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.170317 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.173375 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:10.173380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:10.173384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:10.173416 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:10.175460 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:10.176187 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:10.176198 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:10.176200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:10.176299 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:10.176675 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:10.176681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:10.176684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:10.176964 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:10.177094 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:10.177702 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 14:52:10.177870 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":89}
I20260620 14:52:10.179204 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.163231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:35427 (local address 127.8.221.194:57169)
0620 14:52:10.163791 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:10.163850 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:10.165342 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:10.166123 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:10.166132 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:10.166636 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:10.167276 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.167293 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.168306 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.168313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.169761 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.169776 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.170400 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.170407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.170632 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.171405 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:10.171423 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:10.173170 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:10.175609 (+ 2439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:10.175615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:10.175634 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:10.176058 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:10.176405 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:10.176407 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:10.176408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:10.176558 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:10.177088 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:10.177093 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:10.177415 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 14:52:10.177618 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 14:52:10.178035 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.8.221.195@KRBTEST.COM; defaulting to no policy
I20260620 14:52:10.180780 26448 heartbeater.cc:347] Connected to a master server at 127.8.221.254:35427
I20260620 14:52:10.181094 26448 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:10.181881 26448 heartbeater.cc:511] Master 127.8.221.254:35427 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.195@KRBTEST.COM" created.
I20260620 14:52:10.183552 26123 ts_manager.cc:194] Registered new tserver with Master: 4b857148ce3345e8bb6122fb68185290 (127.8.221.194:41417)
I20260620 14:52:10.184257 26123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:57169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
I20260620 14:52:10.200652 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:35427
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:33501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:10.329392 26453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:10.329731 26453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:10.329782 26453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:10.334676 26453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:10.334765 26453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:10.334872 26453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:52:10.341406 26453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:33501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:35427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:10.342945 26453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:10.344265 26453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:10.356165 26459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:10.356165 26458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:10.356423 26461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:10.356738 26453 server_base.cc:1034] running on GCE node
I20260620 14:52:10.357278 26453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:10.358173 26453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:10.359407 26453 hybrid_clock.cc:648] HybridClock initialized: now 1781967130359389 us; error 31 us; skew 500 ppm
Jun 20 14:52:10 dist-test-slave-65tz krb5kdc[26078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967130, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:10.362567 26453 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:52:10.363575 26453 webserver.cc:460] Webserver started at http://127.8.221.195:43149/ using document root <none> and password file <none>
I20260620 14:52:10.364181 26453 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:10.364248 26453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:10.364431 26453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:10.366822 26453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "b9ac33893356475bb8f4e5b4ab5e1806"
format_stamp: "Formatted at 2026-06-20 14:52:10 on dist-test-slave-65tz"
server_key: "3a099b87eb5acda46fadc3dd491a4fb2"
server_key_iv: "357c76e5f87126f1731497b9d627beb4"
server_key_version: "encryptionkey@0"
I20260620 14:52:10.367369 26453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "b9ac33893356475bb8f4e5b4ab5e1806"
format_stamp: "Formatted at 2026-06-20 14:52:10 on dist-test-slave-65tz"
server_key: "3a099b87eb5acda46fadc3dd491a4fb2"
server_key_iv: "357c76e5f87126f1731497b9d627beb4"
server_key_version: "encryptionkey@0"
I20260620 14:52:10.371706 26453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:52:10.374680 26468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:10.376118 26453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 14:52:10.376279 26453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "b9ac33893356475bb8f4e5b4ab5e1806"
format_stamp: "Formatted at 2026-06-20 14:52:10 on dist-test-slave-65tz"
server_key: "3a099b87eb5acda46fadc3dd491a4fb2"
server_key_iv: "357c76e5f87126f1731497b9d627beb4"
server_key_version: "encryptionkey@0"
I20260620 14:52:10.376408 26453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:10.402339 26453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:10.405777 26453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:10.406040 26453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:10.406701 26453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:10.407634 26453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:10.407708 26453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:10.407786 26453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:10.407832 26453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:10.419488 26453 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:33849
I20260620 14:52:10.419513 26581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:33849 every 8 connection(s)
I20260620 14:52:10.420732 26453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:52:10.429001 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 26453
I20260620 14:52:10.429157 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:52:10.429553 9079 external_mini_cluster.cc:1428] Setting key 1023b1adc170e78e4587e9f763306598
Jun 20 14:52:10 dist-test-slave-65tz krb5kdc[26078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967130, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:52:10.438798 26162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.423331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:32891 (local address 127.8.221.254:35427)
0620 14:52:10.423585 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:10.423592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:10.424735 (+ 1143us) server_negotiation.cc:407] Connection header received
0620 14:52:10.426048 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:10.426052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:10.426117 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:10.426224 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:10.427623 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.428264 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.429629 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.430090 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.430239 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.433966 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:10.433973 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:10.433976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:10.434016 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:10.436203 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:10.436844 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:10.436850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:10.436852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:10.436957 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:10.437355 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:10.437361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:10.437363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:10.437603 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:10.437724 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:10.438442 (+ 718us) server_negotiation.cc:299] Negotiation successful
0620 14:52:10.438591 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 14:52:10.439908 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.423911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:35427 (local address 127.8.221.195:32891)
0620 14:52:10.424533 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:10.424583 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:10.425818 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:10.426458 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:10.426466 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:10.426888 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:10.427436 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.427451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.428421 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.428424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.429487 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.429497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.430329 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.430333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.430537 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.431194 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:10.431218 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:10.433694 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:10.436351 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:10.436360 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:10.436383 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:10.436719 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:10.437065 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:10.437069 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:10.437071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:10.437245 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:10.437736 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:10.437746 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:10.438148 (+ 402us) client_negotiation.cc:769] Sending connection context
0620 14:52:10.438390 (+ 242us) client_negotiation.cc:240] Negotiation successful
0620 14:52:10.438833 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":145,"threads_started":1}
I20260620 14:52:10.441293 26582 heartbeater.cc:347] Connected to a master server at 127.8.221.254:35427
I20260620 14:52:10.441636 26582 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:10.442329 26582 heartbeater.cc:511] Master 127.8.221.254:35427 requested a full tablet report, sending...
I20260620 14:52:10.443713 26123 ts_manager.cc:194] Registered new tserver with Master: b9ac33893356475bb8f4e5b4ab5e1806 (127.8.221.195:33849)
I20260620 14:52:10.444581 26123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:32891
I20260620 14:52:10.444964 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:52:10.456645 26162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.447727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.8.221.254:35427)
0620 14:52:10.447876 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:10.447881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:10.447974 (+ 93us) server_negotiation.cc:407] Connection header received
0620 14:52:10.448076 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:10.448078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:10.448121 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:10.448203 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:10.448982 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.449697 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.450793 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.451049 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.451153 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.452747 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:10.452750 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:10.452753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:10.452794 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:10.454480 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:10.454971 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:10.454975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:10.454976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:10.455105 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:10.455425 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:10.455428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:10.455429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:10.455709 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:10.455836 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:10.456252 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 14:52:10.456451 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:52:10.460268 26123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:52:10.463789 26123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:52:10.484131 26596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.476384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47470 (local address 127.8.221.194:41417)
0620 14:52:10.476830 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:10.476848 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:10.477576 (+ 728us) server_negotiation.cc:407] Connection header received
0620 14:52:10.477719 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:10.477726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:10.478021 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:10.478222 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:10.478998 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.480125 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.482437 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.483285 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.483403 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.483556 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:10.483846 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 14:52:10.483961 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":157,"threads_started":1}
I20260620 14:52:10.484210 26598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.477022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54990 (local address 127.8.221.195:33849)
0620 14:52:10.478710 (+ 1688us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:10.478724 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:10.478754 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:52:10.478837 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:10.478841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:10.479079 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:10.479283 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:10.479556 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.480374 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.482395 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.483281 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.483397 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.483556 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:10.483944 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 14:52:10.484030 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1540,"thread_start_us":334,"threads_started":1}
I20260620 14:52:10.484213 26595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.475901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:41417 (local address 127.0.0.1:47470)
0620 14:52:10.476865 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:10.476927 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:10.477073 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:10.478455 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:10.478461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:10.478493 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:10.478822 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.478833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.480422 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.480426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.482285 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.482296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.483441 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.483453 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.483583 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.483640 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 14:52:10.483794 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 14:52:10.483997 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":172,"spinlock_wait_cycles":14720,"thread_start_us":161,"threads_started":1}
I20260620 14:52:10.484588 26593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.476458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:33849 (local address 127.0.0.1:54990)
0620 14:52:10.477137 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:10.477155 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:10.477283 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:10.479256 (+ 1973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:10.479259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:10.479280 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:10.479428 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.479437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.480539 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.480545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.482240 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.482251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.483632 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.483635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.483714 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.483739 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:52:10.483839 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:52:10.484468 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":635,"thread_start_us":122,"threads_started":1}
I20260620 14:52:10.486562 26597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.475705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:45039 (local address 127.0.0.1:48946)
0620 14:52:10.476865 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:10.476927 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:10.477073 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:10.478453 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:10.478461 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:10.478505 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:10.480421 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.480432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.482148 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.482155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.483101 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.483112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.486063 (+ 2951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.486067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.486180 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.486221 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:52:10.486308 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:52:10.486398 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":1653,"spinlock_wait_cycles":9984,"thread_start_us":106,"threads_started":1}
I20260620 14:52:10.486642 26594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.475834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.8.221.193:45039)
0620 14:52:10.477612 (+ 1778us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:10.477621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:10.477646 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:52:10.477724 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:10.477729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:10.478021 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:10.478222 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:10.480594 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.481994 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.483252 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.484226 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.486047 (+ 1821us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.486231 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:10.486344 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 14:52:10.486473 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1644,"thread_start_us":80,"threads_started":1}
I20260620 14:52:10.488356 26382 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec325b3ba33498b96ace7234e0d1280 (DEFAULT_TABLE table=test-table [id=466ef79f80864cda8e92d9eaa38359ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:10.488513 26516 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec325b3ba33498b96ace7234e0d1280 (DEFAULT_TABLE table=test-table [id=466ef79f80864cda8e92d9eaa38359ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:10.489444 26382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec325b3ba33498b96ace7234e0d1280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:10.489645 26516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec325b3ba33498b96ace7234e0d1280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:10.493432 26248 tablet_service.cc:1467] Processing CreateTablet for tablet 8ec325b3ba33498b96ace7234e0d1280 (DEFAULT_TABLE table=test-table [id=466ef79f80864cda8e92d9eaa38359ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:10.495422 26248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ec325b3ba33498b96ace7234e0d1280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:10.496771 26599 tablet_bootstrap.cc:492] T 8ec325b3ba33498b96ace7234e0d1280 P 4b857148ce3345e8bb6122fb68185290: Bootstrap starting.
I20260620 14:52:10.498842 26600 tablet_bootstrap.cc:492] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806: Bootstrap starting.
I20260620 14:52:10.499604 26599 tablet_bootstrap.cc:654] T 8ec325b3ba33498b96ace7234e0d1280 P 4b857148ce3345e8bb6122fb68185290: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:10.500600 26599 log.cc:824] T 8ec325b3ba33498b96ace7234e0d1280 P 4b857148ce3345e8bb6122fb68185290: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:10.501561 26600 tablet_bootstrap.cc:654] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:10.502763 26600 log.cc:824] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:10.503299 26599 tablet_bootstrap.cc:492] T 8ec325b3ba33498b96ace7234e0d1280 P 4b857148ce3345e8bb6122fb68185290: No bootstrap required, opened a new log
I20260620 14:52:10.503509 26599 ts_tablet_manager.cc:1397] T 8ec325b3ba33498b96ace7234e0d1280 P 4b857148ce3345e8bb6122fb68185290: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 14:52:10.504990 26600 tablet_bootstrap.cc:492] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806: No bootstrap required, opened a new log
I20260620 14:52:10.505200 26603 tablet_bootstrap.cc:492] T 8ec325b3ba33498b96ace7234e0d1280 P 6cb4416263e547b2add25682544cbd19: Bootstrap starting.
I20260620 14:52:10.505272 26600 ts_tablet_manager.cc:1397] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260620 14:52:10.508116 26603 tablet_bootstrap.cc:654] T 8ec325b3ba33498b96ace7234e0d1280 P 6cb4416263e547b2add25682544cbd19: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:10.507962 26599 raft_consensus.cc:348] T 8ec325b3ba33498b96ace7234e0d1280 P 4b857148ce3345e8bb6122fb68185290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb4416263e547b2add25682544cbd19" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45039 } } peers { permanent_uuid: "4b857148ce3345e8bb6122fb68185290" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 41417 } } peers { permanent_uuid: "b9ac33893356475bb8f4e5b4ab5e1806" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33849 } }
I20260620 14:52:10.508312 26599 raft_consensus.cc:374] T 8ec325b3ba33498b96ace7234e0d1280 P 4b857148ce3345e8bb6122fb68185290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:10.508600 26599 raft_consensus.cc:729] T 8ec325b3ba33498b96ace7234e0d1280 P 4b857148ce3345e8bb6122fb68185290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b857148ce3345e8bb6122fb68185290, State: Initialized, Role: FOLLOWER
I20260620 14:52:10.509135 26599 consensus_queue.cc:260] T 8ec325b3ba33498b96ace7234e0d1280 P 4b857148ce3345e8bb6122fb68185290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb4416263e547b2add25682544cbd19" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45039 } } peers { permanent_uuid: "4b857148ce3345e8bb6122fb68185290" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 41417 } } peers { permanent_uuid: "b9ac33893356475bb8f4e5b4ab5e1806" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33849 } }
I20260620 14:52:10.509128 26600 raft_consensus.cc:348] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb4416263e547b2add25682544cbd19" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45039 } } peers { permanent_uuid: "4b857148ce3345e8bb6122fb68185290" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 41417 } } peers { permanent_uuid: "b9ac33893356475bb8f4e5b4ab5e1806" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33849 } }
I20260620 14:52:10.509382 26600 raft_consensus.cc:374] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:10.509454 26600 raft_consensus.cc:729] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ac33893356475bb8f4e5b4ab5e1806, State: Initialized, Role: FOLLOWER
I20260620 14:52:10.509522 26603 log.cc:824] T 8ec325b3ba33498b96ace7234e0d1280 P 6cb4416263e547b2add25682544cbd19: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:10.509934 26600 consensus_queue.cc:260] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb4416263e547b2add25682544cbd19" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45039 } } peers { permanent_uuid: "4b857148ce3345e8bb6122fb68185290" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 41417 } } peers { permanent_uuid: "b9ac33893356475bb8f4e5b4ab5e1806" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33849 } }
I20260620 14:52:10.510411 26448 heartbeater.cc:503] Master 127.8.221.254:35427 was elected leader, sending a full tablet report...
I20260620 14:52:10.510972 26600 ts_tablet_manager.cc:1428] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:52:10.511197 26599 ts_tablet_manager.cc:1428] T 8ec325b3ba33498b96ace7234e0d1280 P 4b857148ce3345e8bb6122fb68185290: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260620 14:52:10.511348 26582 heartbeater.cc:503] Master 127.8.221.254:35427 was elected leader, sending a full tablet report...
I20260620 14:52:10.512578 26603 tablet_bootstrap.cc:492] T 8ec325b3ba33498b96ace7234e0d1280 P 6cb4416263e547b2add25682544cbd19: No bootstrap required, opened a new log
I20260620 14:52:10.512856 26603 ts_tablet_manager.cc:1397] T 8ec325b3ba33498b96ace7234e0d1280 P 6cb4416263e547b2add25682544cbd19: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 14:52:10.518030 26603 raft_consensus.cc:348] T 8ec325b3ba33498b96ace7234e0d1280 P 6cb4416263e547b2add25682544cbd19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb4416263e547b2add25682544cbd19" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45039 } } peers { permanent_uuid: "4b857148ce3345e8bb6122fb68185290" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 41417 } } peers { permanent_uuid: "b9ac33893356475bb8f4e5b4ab5e1806" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33849 } }
I20260620 14:52:10.518353 26603 raft_consensus.cc:374] T 8ec325b3ba33498b96ace7234e0d1280 P 6cb4416263e547b2add25682544cbd19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:10.518450 26603 raft_consensus.cc:729] T 8ec325b3ba33498b96ace7234e0d1280 P 6cb4416263e547b2add25682544cbd19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb4416263e547b2add25682544cbd19, State: Initialized, Role: FOLLOWER
I20260620 14:52:10.518975 26603 consensus_queue.cc:260] T 8ec325b3ba33498b96ace7234e0d1280 P 6cb4416263e547b2add25682544cbd19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb4416263e547b2add25682544cbd19" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45039 } } peers { permanent_uuid: "4b857148ce3345e8bb6122fb68185290" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 41417 } } peers { permanent_uuid: "b9ac33893356475bb8f4e5b4ab5e1806" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33849 } }
I20260620 14:52:10.519794 26314 heartbeater.cc:503] Master 127.8.221.254:35427 was elected leader, sending a full tablet report...
I20260620 14:52:10.520229 26603 ts_tablet_manager.cc:1428] T 8ec325b3ba33498b96ace7234e0d1280 P 6cb4416263e547b2add25682544cbd19: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20260620 14:52:10.628822 26606 raft_consensus.cc:482] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:52:10.629102 26606 raft_consensus.cc:504] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb4416263e547b2add25682544cbd19" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45039 } } peers { permanent_uuid: "4b857148ce3345e8bb6122fb68185290" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 41417 } } peers { permanent_uuid: "b9ac33893356475bb8f4e5b4ab5e1806" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33849 } }
I20260620 14:52:10.630507 26606 leader_election.cc:288] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cb4416263e547b2add25682544cbd19 (127.8.221.193:45039), 4b857148ce3345e8bb6122fb68185290 (127.8.221.194:41417)
I20260620 14:52:10.635612 26596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.631078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:39277 (local address 127.8.221.194:41417)
0620 14:52:10.631234 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:10.631239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:10.631253 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:52:10.631308 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:10.631311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:10.631359 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:10.631437 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:10.631952 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.632659 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.634391 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.634873 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.634969 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.635009 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:10.635400 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 14:52:10.635475 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:52:10.635640 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.630834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:41417 (local address 127.8.221.195:39277)
0620 14:52:10.631027 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:10.631044 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:10.631137 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:10.631571 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:10.631578 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:10.631622 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:10.631818 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.631830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.632773 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.632777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.634257 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.634267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.634998 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.635002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.635164 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.635204 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:52:10.635294 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:52:10.635405 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":40}
I20260620 14:52:10.635908 26608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.630871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:45039 (local address 127.8.221.195:59969)
0620 14:52:10.631423 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:10.631437 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:10.631582 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:10.631937 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:10.631940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:10.631960 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:10.632096 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.632103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.633327 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.633330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.634494 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.634505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.635417 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.635420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.635520 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.635555 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:52:10.635642 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:52:10.635715 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":54,"thread_start_us":259,"threads_started":1}
I20260620 14:52:10.635931 26594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.631297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:59969 (local address 127.8.221.193:45039)
0620 14:52:10.631479 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:10.631487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:10.631508 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:52:10.631718 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:10.631721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:10.631786 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:10.631896 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:10.632219 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.633171 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.634631 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.635280 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.635400 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.635457 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:10.635680 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 14:52:10.635771 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:52:10.636368 26402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec325b3ba33498b96ace7234e0d1280" candidate_uuid: "b9ac33893356475bb8f4e5b4ab5e1806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b857148ce3345e8bb6122fb68185290" is_pre_election: true
I20260620 14:52:10.636695 26402 raft_consensus.cc:2454] T 8ec325b3ba33498b96ace7234e0d1280 P 4b857148ce3345e8bb6122fb68185290 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9ac33893356475bb8f4e5b4ab5e1806 in term 0.
I20260620 14:52:10.636830 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec325b3ba33498b96ace7234e0d1280" candidate_uuid: "b9ac33893356475bb8f4e5b4ab5e1806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cb4416263e547b2add25682544cbd19" is_pre_election: true
I20260620 14:52:10.637218 26268 raft_consensus.cc:2454] T 8ec325b3ba33498b96ace7234e0d1280 P 6cb4416263e547b2add25682544cbd19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9ac33893356475bb8f4e5b4ab5e1806 in term 0.
I20260620 14:52:10.637267 26469 leader_election.cc:302] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b857148ce3345e8bb6122fb68185290, b9ac33893356475bb8f4e5b4ab5e1806; no voters:
I20260620 14:52:10.637622 26606 raft_consensus.cc:2781] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:52:10.637744 26606 raft_consensus.cc:482] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:52:10.637840 26606 raft_consensus.cc:3037] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:10.638953 26606 raft_consensus.cc:504] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb4416263e547b2add25682544cbd19" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45039 } } peers { permanent_uuid: "4b857148ce3345e8bb6122fb68185290" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 41417 } } peers { permanent_uuid: "b9ac33893356475bb8f4e5b4ab5e1806" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33849 } }
I20260620 14:52:10.639343 26606 leader_election.cc:288] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [CANDIDATE]: Term 1 election: Requested vote from peers 6cb4416263e547b2add25682544cbd19 (127.8.221.193:45039), 4b857148ce3345e8bb6122fb68185290 (127.8.221.194:41417)
I20260620 14:52:10.639827 26402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec325b3ba33498b96ace7234e0d1280" candidate_uuid: "b9ac33893356475bb8f4e5b4ab5e1806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b857148ce3345e8bb6122fb68185290"
I20260620 14:52:10.639978 26268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ec325b3ba33498b96ace7234e0d1280" candidate_uuid: "b9ac33893356475bb8f4e5b4ab5e1806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cb4416263e547b2add25682544cbd19"
I20260620 14:52:10.640022 26402 raft_consensus.cc:3037] T 8ec325b3ba33498b96ace7234e0d1280 P 4b857148ce3345e8bb6122fb68185290 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:10.640198 26268 raft_consensus.cc:3037] T 8ec325b3ba33498b96ace7234e0d1280 P 6cb4416263e547b2add25682544cbd19 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:10.641387 26268 raft_consensus.cc:2454] T 8ec325b3ba33498b96ace7234e0d1280 P 6cb4416263e547b2add25682544cbd19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9ac33893356475bb8f4e5b4ab5e1806 in term 1.
I20260620 14:52:10.641590 26402 raft_consensus.cc:2454] T 8ec325b3ba33498b96ace7234e0d1280 P 4b857148ce3345e8bb6122fb68185290 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9ac33893356475bb8f4e5b4ab5e1806 in term 1.
I20260620 14:52:10.641744 26470 leader_election.cc:302] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cb4416263e547b2add25682544cbd19, b9ac33893356475bb8f4e5b4ab5e1806; no voters:
I20260620 14:52:10.642102 26606 raft_consensus.cc:2781] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:10.642300 26606 raft_consensus.cc:686] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [term 1 LEADER]: Becoming Leader. State: Replica: b9ac33893356475bb8f4e5b4ab5e1806, State: Running, Role: LEADER
I20260620 14:52:10.642686 26606 consensus_queue.cc:237] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb4416263e547b2add25682544cbd19" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45039 } } peers { permanent_uuid: "4b857148ce3345e8bb6122fb68185290" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 41417 } } peers { permanent_uuid: "b9ac33893356475bb8f4e5b4ab5e1806" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33849 } }
I20260620 14:52:10.646039 26123 catalog_manager.cc:5495] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9ac33893356475bb8f4e5b4ab5e1806 (127.8.221.195). New cstate: current_term: 1 leader_uuid: "b9ac33893356475bb8f4e5b4ab5e1806" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cb4416263e547b2add25682544cbd19" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b857148ce3345e8bb6122fb68185290" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 41417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9ac33893356475bb8f4e5b4ab5e1806" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33849 } health_report { overall_health: HEALTHY } } }
W20260620 14:52:10.661919 26449 tablet.cc:2369] T 8ec325b3ba33498b96ace7234e0d1280 P 4b857148ce3345e8bb6122fb68185290: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:52:10.671994 26583 tablet.cc:2369] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:52:10.677728 26598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.674106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54994 (local address 127.8.221.195:33849)
0620 14:52:10.674268 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:10.674273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:10.674288 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:52:10.674365 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:10.674367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:10.674422 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:10.674583 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:10.674850 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.675363 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.676264 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.676477 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.676588 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.676888 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:10.677299 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:10.677429 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:10.677491 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 14:52:10.677543 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 14:52:10.680382 26315 tablet.cc:2369] T 8ec325b3ba33498b96ace7234e0d1280 P 6cb4416263e547b2add25682544cbd19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:52:10.684921 26268 raft_consensus.cc:1261] T 8ec325b3ba33498b96ace7234e0d1280 P 6cb4416263e547b2add25682544cbd19 [term 1 FOLLOWER]: Refusing update from remote peer b9ac33893356475bb8f4e5b4ab5e1806: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:10.684921 26402 raft_consensus.cc:1261] T 8ec325b3ba33498b96ace7234e0d1280 P 4b857148ce3345e8bb6122fb68185290 [term 1 FOLLOWER]: Refusing update from remote peer b9ac33893356475bb8f4e5b4ab5e1806: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:10.685945 26606 consensus_queue.cc:1035] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b857148ce3345e8bb6122fb68185290" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 41417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:10.686218 26609 consensus_queue.cc:1035] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cb4416263e547b2add25682544cbd19" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:10.697182 26614 mvcc.cc:205] Tried to move back new op lower bound from 7298937367275937792 to 7298937367115280384. Current Snapshot: MvccSnapshot[applied={T|T < 7298937367275937792}]
I20260620 14:52:10.706022 26123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:10.706261 26123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:10.709154 26123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a578fc007e349dc9b209955f0206752: Sending DeleteTablet for 3 replicas of tablet 8ec325b3ba33498b96ace7234e0d1280
I20260620 14:52:10.710114 26516 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec325b3ba33498b96ace7234e0d1280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:10 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:54990
I20260620 14:52:10.710199 26382 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec325b3ba33498b96ace7234e0d1280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:10 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:47470
I20260620 14:52:10.710222 26248 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ec325b3ba33498b96ace7234e0d1280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:10 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:48946
I20260620 14:52:10.710832 26622 tablet_replica.cc:330] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806: stopping tablet replica
I20260620 14:52:10.710984 26623 tablet_replica.cc:330] T 8ec325b3ba33498b96ace7234e0d1280 P 6cb4416263e547b2add25682544cbd19: stopping tablet replica
I20260620 14:52:10.711184 26622 raft_consensus.cc:2229] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:52:10.711356 26623 raft_consensus.cc:2229] T 8ec325b3ba33498b96ace7234e0d1280 P 6cb4416263e547b2add25682544cbd19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:10.711565 26622 raft_consensus.cc:2258] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:10.711933 26621 tablet_replica.cc:330] T 8ec325b3ba33498b96ace7234e0d1280 P 4b857148ce3345e8bb6122fb68185290: stopping tablet replica
I20260620 14:52:10.711987 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 26185
I20260620 14:52:10.712297 26621 raft_consensus.cc:2229] T 8ec325b3ba33498b96ace7234e0d1280 P 4b857148ce3345e8bb6122fb68185290 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:10.712617 26621 raft_consensus.cc:2258] T 8ec325b3ba33498b96ace7234e0d1280 P 4b857148ce3345e8bb6122fb68185290 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:10.713721 26622 ts_tablet_manager.cc:1905] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:10.717103 26622 ts_tablet_manager.cc:1918] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:10.717235 26622 log.cc:1196] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/8ec325b3ba33498b96ace7234e0d1280
I20260620 14:52:10.717592 26622 ts_tablet_manager.cc:1939] T 8ec325b3ba33498b96ace7234e0d1280 P b9ac33893356475bb8f4e5b4ab5e1806: Deleting consensus metadata
I20260620 14:52:10.719110 26109 catalog_manager.cc:4841] TS b9ac33893356475bb8f4e5b4ab5e1806 (127.8.221.195:33849): tablet 8ec325b3ba33498b96ace7234e0d1280 (table test-table [id=466ef79f80864cda8e92d9eaa38359ec]) successfully deleted
I20260620 14:52:10.719208 26621 ts_tablet_manager.cc:1905] T 8ec325b3ba33498b96ace7234e0d1280 P 4b857148ce3345e8bb6122fb68185290: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:10.721717 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 26319
W20260620 14:52:10.721879 26108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.221.193:45039 (error 108)
I20260620 14:52:10.723413 26597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.723094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:45039 (local address 127.0.0.1:48952)
0620 14:52:10.723247 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:10.723320 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.193:45039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 14:52:10.723559 26108 catalog_manager.cc:4584] TS 6cb4416263e547b2add25682544cbd19 (127.8.221.193:45039): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ec325b3ba33498b96ace7234e0d1280: Network error: Client connection negotiation failed: client connection to 127.8.221.193:45039: connect: Connection refused (error 111)
I20260620 14:52:10.730572 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 26453
I20260620 14:52:10.730675 26597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.730373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:41417 (local address 127.0.0.1:47482)
0620 14:52:10.730514 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:10.730574 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.194:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 14:52:10.737784 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 26091
2026-06-20T14:52:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2571 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T14:52:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:10Z Disabled control of system clock
I20260620 14:52:10.773720 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:46683
--webserver_interface=127.8.221.254
--webserver_port=0
--builtin_ntp_servers=127.8.221.212:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:46683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:52:10.905077 26628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:10.905457 26628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:10.905499 26628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:10.909960 26628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:10.910050 26628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:10.910066 26628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:10.910085 26628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:10.910101 26628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:10.915397 26628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:46683
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:46683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:10.916777 26628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:10.918439 26628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:10.928194 26634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:10.928373 26633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:10.928210 26636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:10.928889 26628 server_base.cc:1034] running on GCE node
I20260620 14:52:10.929584 26628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:10.930919 26628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:10.932138 26628 hybrid_clock.cc:648] HybridClock initialized: now 1781967130932103 us; error 46 us; skew 500 ppm
I20260620 14:52:10.934283 26628 webserver.cc:460] Webserver started at http://127.8.221.254:34859/ using document root <none> and password file <none>
I20260620 14:52:10.934916 26628 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:10.935014 26628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:10.935238 26628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:10.937551 26628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "66ee9253e44e4272b6280aa673deab0e"
format_stamp: "Formatted at 2026-06-20 14:52:10 on dist-test-slave-65tz"
server_key: "7ef2750938c2989f0aee9d6c1ad7279c"
server_key_iv: "9ff13cbeae98d1f74588517c4bc21be4"
server_key_version: "encryptionkey@0"
I20260620 14:52:10.938187 26628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "66ee9253e44e4272b6280aa673deab0e"
format_stamp: "Formatted at 2026-06-20 14:52:10 on dist-test-slave-65tz"
server_key: "7ef2750938c2989f0aee9d6c1ad7279c"
server_key_iv: "9ff13cbeae98d1f74588517c4bc21be4"
server_key_version: "encryptionkey@0"
I20260620 14:52:10.942448 26628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:52:10.945439 26642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:10.947162 26628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:10.947366 26628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "66ee9253e44e4272b6280aa673deab0e"
format_stamp: "Formatted at 2026-06-20 14:52:10 on dist-test-slave-65tz"
server_key: "7ef2750938c2989f0aee9d6c1ad7279c"
server_key_iv: "9ff13cbeae98d1f74588517c4bc21be4"
server_key_version: "encryptionkey@0"
I20260620 14:52:10.947511 26628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:10.987041 26628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:10.987855 26628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:10.988106 26628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:10.996685 26628 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:46683
I20260620 14:52:10.996759 26694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:46683 every 8 connection(s)
I20260620 14:52:10.997831 26628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:11.001269 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 26628
I20260620 14:52:11.001444 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:11.001924 9079 external_mini_cluster.cc:1428] Setting key 54d85f2312e8b2b520c4b74630fd0db6
I20260620 14:52:11.002292 26695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:11.011437 26695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e: Bootstrap starting.
I20260620 14:52:11.012600 26698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:11.003646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46846 (local address 127.8.221.254:46683)
0620 14:52:11.004354 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:11.004372 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:11.004411 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:52:11.005120 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:11.005136 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:11.005535 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:11.005914 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:11.006501 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.007535 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.008860 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.009396 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.009537 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:11.010669 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:11.010674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:11.010689 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:11.011128 (+ 439us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:11.011189 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:11.011209 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:11.011326 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:11.011495 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 14:52:11.011812 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":160,"threads_started":1}
I20260620 14:52:11.014607 26695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:11.015657 26695 log.cc:824] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:11.018428 26695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e: No bootstrap required, opened a new log
I20260620 14:52:11.023161 26695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee9253e44e4272b6280aa673deab0e" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 46683 } }
I20260620 14:52:11.023509 26695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:11.023561 26695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ee9253e44e4272b6280aa673deab0e, State: Initialized, Role: FOLLOWER
I20260620 14:52:11.024125 26695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee9253e44e4272b6280aa673deab0e" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 46683 } }
I20260620 14:52:11.024277 26695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:11.024326 26695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:11.024421 26695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:11.025668 26695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee9253e44e4272b6280aa673deab0e" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 46683 } }
I20260620 14:52:11.026197 26695 leader_election.cc:302] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66ee9253e44e4272b6280aa673deab0e; no voters:
I20260620 14:52:11.026744 26695 leader_election.cc:288] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:11.026988 26700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:11.027308 26700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e [term 1 LEADER]: Becoming Leader. State: Replica: 66ee9253e44e4272b6280aa673deab0e, State: Running, Role: LEADER
I20260620 14:52:11.027717 26700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee9253e44e4272b6280aa673deab0e" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 46683 } }
I20260620 14:52:11.028405 26695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:11.029668 26701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66ee9253e44e4272b6280aa673deab0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee9253e44e4272b6280aa673deab0e" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 46683 } } }
I20260620 14:52:11.029876 26701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:11.030211 26702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66ee9253e44e4272b6280aa673deab0e. Latest consensus state: current_term: 1 leader_uuid: "66ee9253e44e4272b6280aa673deab0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee9253e44e4272b6280aa673deab0e" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 46683 } } }
I20260620 14:52:11.030320 26702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:11.030413 26709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:11.034518 26709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:11.042147 26709 catalog_manager.cc:1347] Generated new cluster ID: f102f32df6fd485486685c663f3f90dd
I20260620 14:52:11.042234 26709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:11.075500 26709 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:11.077131 26709 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:11.090327 26709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e: Generated new TSK 0
I20260620 14:52:11.091270 26709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:11.104429 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:46683
--builtin_ntp_servers=127.8.221.212:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:52:11.238070 26719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:11.238430 26719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:11.238505 26719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:11.242884 26719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:11.242966 26719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:11.243078 26719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:52:11.248019 26719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:46683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:11.249361 26719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:11.251174 26719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:11.262722 26724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:11.262722 26727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:11.262759 26725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:11.263218 26719 server_base.cc:1034] running on GCE node
I20260620 14:52:11.263792 26719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:11.264600 26719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:11.265868 26719 hybrid_clock.cc:648] HybridClock initialized: now 1781967131265841 us; error 43 us; skew 500 ppm
I20260620 14:52:11.268153 26719 webserver.cc:460] Webserver started at http://127.8.221.193:40563/ using document root <none> and password file <none>
I20260620 14:52:11.268877 26719 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:11.268977 26719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:11.269248 26719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:11.271818 26719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "ce5edaf4bbe74d88baa835539c12b542"
format_stamp: "Formatted at 2026-06-20 14:52:11 on dist-test-slave-65tz"
server_key: "ff75b643c1dc4c4ff99374ab0d8cbd7d"
server_key_iv: "f4a55e7e47a75b19ae5fbd6742f73938"
server_key_version: "encryptionkey@0"
I20260620 14:52:11.272491 26719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "ce5edaf4bbe74d88baa835539c12b542"
format_stamp: "Formatted at 2026-06-20 14:52:11 on dist-test-slave-65tz"
server_key: "ff75b643c1dc4c4ff99374ab0d8cbd7d"
server_key_iv: "f4a55e7e47a75b19ae5fbd6742f73938"
server_key_version: "encryptionkey@0"
I20260620 14:52:11.277098 26719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:52:11.280308 26733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:11.282318 26719 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:52:11.282507 26719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "ce5edaf4bbe74d88baa835539c12b542"
format_stamp: "Formatted at 2026-06-20 14:52:11 on dist-test-slave-65tz"
server_key: "ff75b643c1dc4c4ff99374ab0d8cbd7d"
server_key_iv: "f4a55e7e47a75b19ae5fbd6742f73938"
server_key_version: "encryptionkey@0"
I20260620 14:52:11.282673 26719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:11.301255 26719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:11.302325 26719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:11.302567 26719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:11.303417 26719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:11.304648 26719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:11.304724 26719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:11.304800 26719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:11.304852 26719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:11.316834 26719 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:43017
I20260620 14:52:11.316897 26846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:43017 every 8 connection(s)
I20260620 14:52:11.318073 26719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:52:11.323474 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 26719
I20260620 14:52:11.323665 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:52:11.324021 9079 external_mini_cluster.cc:1428] Setting key d55f9c69ebf66665d3b95e8127a69757
I20260620 14:52:11.326769 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:46683
--builtin_ntp_servers=127.8.221.212:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:52:11.330438 26698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:11.320378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:57473 (local address 127.8.221.254:46683)
0620 14:52:11.320587 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:11.320591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:11.321881 (+ 1290us) server_negotiation.cc:407] Connection header received
0620 14:52:11.323475 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:11.323478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:11.323545 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:11.323658 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:11.324818 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.325556 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.327579 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.327799 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.327914 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:11.329711 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:11.329718 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:11.329723 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:11.329740 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:11.329772 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:11.329784 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:11.329903 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:11.330073 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 14:52:11.330150 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 14:52:11.331405 26849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:11.320854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:46683 (local address 127.8.221.193:57473)
0620 14:52:11.321660 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:11.321710 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:11.323244 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:11.323876 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:11.323883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:11.323974 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:11.324615 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:11.324636 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.325727 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.325732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:11.327408 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:11.327428 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.328007 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.328014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:11.328280 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:11.329335 (+ 1055us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:52:11.329337 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:11.329415 (+ 78us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:52:11.329524 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:11.329904 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:11.329912 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:11.329917 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:52:11.330038 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 14:52:11.330359 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":125,"threads_started":1}
I20260620 14:52:11.333030 26847 heartbeater.cc:347] Connected to a master server at 127.8.221.254:46683
I20260620 14:52:11.333402 26847 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:11.334244 26847 heartbeater.cc:511] Master 127.8.221.254:46683 requested a full tablet report, sending...
I20260620 14:52:11.336241 26659 ts_manager.cc:194] Registered new tserver with Master: ce5edaf4bbe74d88baa835539c12b542 (127.8.221.193:43017)
I20260620 14:52:11.337694 26659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.221.193:57473
W20260620 14:52:11.450155 26850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:11.450505 26850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:11.450588 26850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:11.455268 26850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:11.455354 26850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:11.455422 26850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:52:11.460939 26850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:46683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:11.462479 26850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:11.464329 26850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:11.475808 26856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:11.476133 26850 server_base.cc:1034] running on GCE node
W20260620 14:52:11.475821 26858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:11.475824 26855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:11.476739 26850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:11.477510 26850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:11.478698 26850 hybrid_clock.cc:648] HybridClock initialized: now 1781967131478688 us; error 67 us; skew 500 ppm
I20260620 14:52:11.480759 26850 webserver.cc:460] Webserver started at http://127.8.221.194:42017/ using document root <none> and password file <none>
I20260620 14:52:11.481407 26850 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:11.481472 26850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:11.481709 26850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:11.483901 26850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "61ab904efffc468b92f4c1b91d42e726"
format_stamp: "Formatted at 2026-06-20 14:52:11 on dist-test-slave-65tz"
server_key: "dd2627d52acb8f063070a39d415f2e5c"
server_key_iv: "a68b4299fa73da66ef780ac6f9186750"
server_key_version: "encryptionkey@0"
I20260620 14:52:11.484468 26850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "61ab904efffc468b92f4c1b91d42e726"
format_stamp: "Formatted at 2026-06-20 14:52:11 on dist-test-slave-65tz"
server_key: "dd2627d52acb8f063070a39d415f2e5c"
server_key_iv: "a68b4299fa73da66ef780ac6f9186750"
server_key_version: "encryptionkey@0"
I20260620 14:52:11.489007 26850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 14:52:11.492048 26864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:11.493587 26850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:52:11.493701 26850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "61ab904efffc468b92f4c1b91d42e726"
format_stamp: "Formatted at 2026-06-20 14:52:11 on dist-test-slave-65tz"
server_key: "dd2627d52acb8f063070a39d415f2e5c"
server_key_iv: "a68b4299fa73da66ef780ac6f9186750"
server_key_version: "encryptionkey@0"
I20260620 14:52:11.493881 26850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:11.527855 26850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:11.528579 26850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:11.528820 26850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:11.529496 26850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:11.530735 26850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:11.530786 26850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:11.530872 26850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:11.530910 26850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:11.541173 26850 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:38383
I20260620 14:52:11.541261 26977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:38383 every 8 connection(s)
I20260620 14:52:11.542229 26850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:52:11.545871 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 26850
I20260620 14:52:11.546030 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:52:11.546466 9079 external_mini_cluster.cc:1428] Setting key f70c0dff00e1a52c1a5a89b76b750476
I20260620 14:52:11.548823 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:46683
--builtin_ntp_servers=127.8.221.212:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:52:11.554171 26698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:11.544708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:35851 (local address 127.8.221.254:46683)
0620 14:52:11.544882 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:11.544886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:11.546329 (+ 1443us) server_negotiation.cc:407] Connection header received
0620 14:52:11.548066 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:11.548070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:11.548134 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:11.548235 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:11.549396 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.550349 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.551752 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.552111 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.552208 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:11.553579 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:11.553583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:11.553585 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:11.553599 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:11.553625 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:11.553633 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:11.553716 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:11.553930 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 14:52:11.554016 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:52:11.555011 26980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:11.545034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:46683 (local address 127.8.221.194:35851)
0620 14:52:11.546077 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:11.546150 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:11.547809 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:11.548476 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:11.548485 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:11.548590 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:11.549228 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:11.549244 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.550509 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.550512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:11.551599 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:11.551612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.552347 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.552351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:11.552512 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:11.553262 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:52:11.553263 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:11.553326 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:52:11.553396 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:11.553730 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:11.553740 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:11.553753 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:52:11.553920 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 14:52:11.554204 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":142,"threads_started":1}
I20260620 14:52:11.556243 26978 heartbeater.cc:347] Connected to a master server at 127.8.221.254:46683
I20260620 14:52:11.556505 26978 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:11.557032 26978 heartbeater.cc:511] Master 127.8.221.254:46683 requested a full tablet report, sending...
I20260620 14:52:11.558259 26659 ts_manager.cc:194] Registered new tserver with Master: 61ab904efffc468b92f4c1b91d42e726 (127.8.221.194:38383)
I20260620 14:52:11.559413 26659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.221.194:35851
W20260620 14:52:11.679383 26981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:11.679695 26981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:11.679737 26981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:11.683961 26981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:11.684029 26981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:11.684095 26981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:52:11.689414 26981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:46683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:11.690864 26981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:11.692545 26981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:11.704247 26989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:11.704276 26986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:11.704280 26987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:11.704859 26981 server_base.cc:1034] running on GCE node
I20260620 14:52:11.705410 26981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:11.706367 26981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:11.707610 26981 hybrid_clock.cc:648] HybridClock initialized: now 1781967131707531 us; error 78 us; skew 500 ppm
I20260620 14:52:11.710183 26981 webserver.cc:460] Webserver started at http://127.8.221.195:37717/ using document root <none> and password file <none>
I20260620 14:52:11.710886 26981 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:11.710954 26981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:11.711180 26981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:11.713637 26981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "907eed367beb47bc9d8e4a30dcdc710b"
format_stamp: "Formatted at 2026-06-20 14:52:11 on dist-test-slave-65tz"
server_key: "3291761cf88e1e0a169ab9f0dff53f03"
server_key_iv: "cfb06889ffdab62cb1b3201f1e97c296"
server_key_version: "encryptionkey@0"
I20260620 14:52:11.714355 26981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "907eed367beb47bc9d8e4a30dcdc710b"
format_stamp: "Formatted at 2026-06-20 14:52:11 on dist-test-slave-65tz"
server_key: "3291761cf88e1e0a169ab9f0dff53f03"
server_key_iv: "cfb06889ffdab62cb1b3201f1e97c296"
server_key_version: "encryptionkey@0"
I20260620 14:52:11.719319 26981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:52:11.722306 26995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:11.723867 26981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:52:11.724006 26981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "907eed367beb47bc9d8e4a30dcdc710b"
format_stamp: "Formatted at 2026-06-20 14:52:11 on dist-test-slave-65tz"
server_key: "3291761cf88e1e0a169ab9f0dff53f03"
server_key_iv: "cfb06889ffdab62cb1b3201f1e97c296"
server_key_version: "encryptionkey@0"
I20260620 14:52:11.724121 26981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:11.747843 26981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:11.748740 26981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:11.748922 26981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:11.749615 26981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:11.750919 26981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:11.750973 26981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:11.751048 26981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:11.751084 26981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:11.763019 26981 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:44021
I20260620 14:52:11.763098 27108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:44021 every 8 connection(s)
I20260620 14:52:11.764268 26981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:52:11.768141 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 26981
I20260620 14:52:11.768291 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:52:11.768612 9079 external_mini_cluster.cc:1428] Setting key 18bb5c36d2a434203cb093daf5df1529
I20260620 14:52:11.774981 26698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:11.766506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:34951 (local address 127.8.221.254:46683)
0620 14:52:11.766724 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:11.766729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:11.767910 (+ 1181us) server_negotiation.cc:407] Connection header received
0620 14:52:11.769596 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:11.769599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:11.769666 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:11.769756 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:11.771070 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.771603 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.772913 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.773134 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.773250 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:11.774519 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:11.774523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:11.774525 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:11.774537 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:11.774557 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:11.774565 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:11.774634 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:11.774804 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 14:52:11.774865 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:52:11.776048 27111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:11.766893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:46683 (local address 127.8.221.195:34951)
0620 14:52:11.767706 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:11.767756 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:11.769385 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:11.770029 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:11.770040 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:11.770165 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:11.770900 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:11.770926 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.771730 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.771733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:11.772749 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:11.772758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.773312 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.773315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:11.773468 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:11.774231 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:52:11.774233 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:11.774280 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:52:11.774356 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:11.774661 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:11.774668 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:11.774672 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:52:11.774781 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:52:11.775078 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":139,"threads_started":1}
I20260620 14:52:11.777734 27109 heartbeater.cc:347] Connected to a master server at 127.8.221.254:46683
I20260620 14:52:11.778105 27109 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:11.778813 27109 heartbeater.cc:511] Master 127.8.221.254:46683 requested a full tablet report, sending...
I20260620 14:52:11.780206 26659 ts_manager.cc:194] Registered new tserver with Master: 907eed367beb47bc9d8e4a30dcdc710b (127.8.221.195:44021)
I20260620 14:52:11.780961 26659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.221.195:34951
I20260620 14:52:11.783545 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:52:11.790318 26698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:11.785960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46852 (local address 127.8.221.254:46683)
0620 14:52:11.786115 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:11.786118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:11.786482 (+ 364us) server_negotiation.cc:407] Connection header received
0620 14:52:11.786673 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:11.786676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:11.786731 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:11.786822 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:11.787110 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.787770 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.788635 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.788856 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.788977 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:11.789863 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:11.789867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:11.789870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:11.789891 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:11.789908 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:11.789915 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:11.790013 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:11.790107 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 14:52:11.790161 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:52:11.794149 26659 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:52:11.798422 26659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:52:11.814785 27121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:11.808687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39408 (local address 127.8.221.194:38383)
0620 14:52:11.809951 (+ 1264us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:11.809963 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:11.809984 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:52:11.810055 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:11.810059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:11.810193 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:11.810704 (+ 511us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:11.810825 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.811721 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.813159 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.814007 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.814128 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:11.814327 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:11.814514 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 14:52:11.814610 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1169,"thread_start_us":133,"threads_started":1}
I20260620 14:52:11.814777 27120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:11.808531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:38383 (local address 127.0.0.1:39408)
0620 14:52:11.809036 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:11.809097 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:11.809453 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:11.810382 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:11.810388 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:11.810411 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:11.810552 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:11.810560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.811828 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.811834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:11.813016 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:11.813026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.814140 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.814143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:11.814250 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:11.814332 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 14:52:11.814463 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 14:52:11.814606 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":153,"threads_started":1}
I20260620 14:52:11.815681 27122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:11.808680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:44021 (local address 127.0.0.1:46720)
0620 14:52:11.809630 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:11.809647 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:11.809749 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:11.810497 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:11.810512 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:11.810528 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:11.810608 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:11.810616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.812430 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.812433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:11.813541 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:11.813550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.815133 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.815137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:11.815243 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:11.815276 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:52:11.815375 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:52:11.815474 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"mutex_wait_us":5,"thread_start_us":135,"threads_started":1}
I20260620 14:52:11.816289 27123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:11.809245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46720 (local address 127.8.221.195:44021)
0620 14:52:11.809698 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:11.809718 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:11.809743 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:52:11.809966 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:11.809974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:11.810191 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:11.810356 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:11.810773 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.812272 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.813700 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.814703 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.814822 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:11.815042 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:11.815937 (+ 895us) server_negotiation.cc:299] Negotiation successful
0620 14:52:11.816071 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":142,"threads_started":1}
I20260620 14:52:11.816921 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:11.809934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45488 (local address 127.8.221.193:43017)
0620 14:52:11.810585 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:11.810596 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:11.810637 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:52:11.810768 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:11.810773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:11.811007 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:11.811198 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:11.811437 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.812827 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.814562 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.815449 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.815564 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:11.815769 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:11.816591 (+ 822us) server_negotiation.cc:299] Negotiation successful
0620 14:52:11.816762 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":128,"threads_started":1}
I20260620 14:52:11.816337 27124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:11.809474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:43017 (local address 127.0.0.1:45488)
0620 14:52:11.810402 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:11.810418 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:11.810522 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:11.811189 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:11.811193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:11.811211 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:11.811298 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:11.811304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.812967 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.812971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:11.814388 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:11.814400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.815589 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.815593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:11.815691 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:11.815716 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:52:11.816098 (+ 382us) client_negotiation.cc:240] Negotiation successful
0620 14:52:11.816200 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":151,"threads_started":1}
I20260620 14:52:11.819087 26912 tablet_service.cc:1467] Processing CreateTablet for tablet ecf92d7448b44f67bf8848c4268d50d7 (DEFAULT_TABLE table=test-table [id=02cac29d69f049989e916e6f6d7cbbc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:11.820473 26912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecf92d7448b44f67bf8848c4268d50d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:11.820252 26781 tablet_service.cc:1467] Processing CreateTablet for tablet ecf92d7448b44f67bf8848c4268d50d7 (DEFAULT_TABLE table=test-table [id=02cac29d69f049989e916e6f6d7cbbc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:11.820253 27043 tablet_service.cc:1467] Processing CreateTablet for tablet ecf92d7448b44f67bf8848c4268d50d7 (DEFAULT_TABLE table=test-table [id=02cac29d69f049989e916e6f6d7cbbc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:11.821424 26781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecf92d7448b44f67bf8848c4268d50d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:11.821424 27043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecf92d7448b44f67bf8848c4268d50d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:11.829615 27126 tablet_bootstrap.cc:492] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726: Bootstrap starting.
I20260620 14:52:11.831478 27127 tablet_bootstrap.cc:492] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b: Bootstrap starting.
I20260620 14:52:11.832707 27126 tablet_bootstrap.cc:654] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:11.833866 27126 log.cc:824] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:11.834069 27127 tablet_bootstrap.cc:654] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:11.834188 27128 tablet_bootstrap.cc:492] T ecf92d7448b44f67bf8848c4268d50d7 P ce5edaf4bbe74d88baa835539c12b542: Bootstrap starting.
I20260620 14:52:11.835021 27127 log.cc:824] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:11.836378 27128 tablet_bootstrap.cc:654] T ecf92d7448b44f67bf8848c4268d50d7 P ce5edaf4bbe74d88baa835539c12b542: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:11.836386 27126 tablet_bootstrap.cc:492] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726: No bootstrap required, opened a new log
I20260620 14:52:11.836654 27126 ts_tablet_manager.cc:1397] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260620 14:52:11.837184 27128 log.cc:824] T ecf92d7448b44f67bf8848c4268d50d7 P ce5edaf4bbe74d88baa835539c12b542: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:11.837441 27127 tablet_bootstrap.cc:492] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b: No bootstrap required, opened a new log
I20260620 14:52:11.837666 27127 ts_tablet_manager.cc:1397] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:52:11.839787 27128 tablet_bootstrap.cc:492] T ecf92d7448b44f67bf8848c4268d50d7 P ce5edaf4bbe74d88baa835539c12b542: No bootstrap required, opened a new log
I20260620 14:52:11.840116 27128 ts_tablet_manager.cc:1397] T ecf92d7448b44f67bf8848c4268d50d7 P ce5edaf4bbe74d88baa835539c12b542: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:52:11.841274 27126 raft_consensus.cc:348] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ab904efffc468b92f4c1b91d42e726" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38383 } } peers { permanent_uuid: "ce5edaf4bbe74d88baa835539c12b542" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43017 } } peers { permanent_uuid: "907eed367beb47bc9d8e4a30dcdc710b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44021 } }
I20260620 14:52:11.841630 27126 raft_consensus.cc:374] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:11.841687 27126 raft_consensus.cc:729] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61ab904efffc468b92f4c1b91d42e726, State: Initialized, Role: FOLLOWER
I20260620 14:52:11.842285 27126 consensus_queue.cc:260] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ab904efffc468b92f4c1b91d42e726" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38383 } } peers { permanent_uuid: "ce5edaf4bbe74d88baa835539c12b542" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43017 } } peers { permanent_uuid: "907eed367beb47bc9d8e4a30dcdc710b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44021 } }
I20260620 14:52:11.842687 27127 raft_consensus.cc:348] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ab904efffc468b92f4c1b91d42e726" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38383 } } peers { permanent_uuid: "ce5edaf4bbe74d88baa835539c12b542" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43017 } } peers { permanent_uuid: "907eed367beb47bc9d8e4a30dcdc710b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44021 } }
I20260620 14:52:11.843027 27127 raft_consensus.cc:374] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:11.843127 27127 raft_consensus.cc:729] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907eed367beb47bc9d8e4a30dcdc710b, State: Initialized, Role: FOLLOWER
I20260620 14:52:11.843245 26978 heartbeater.cc:503] Master 127.8.221.254:46683 was elected leader, sending a full tablet report...
I20260620 14:52:11.843643 27126 ts_tablet_manager.cc:1428] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260620 14:52:11.843713 27127 consensus_queue.cc:260] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ab904efffc468b92f4c1b91d42e726" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38383 } } peers { permanent_uuid: "ce5edaf4bbe74d88baa835539c12b542" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43017 } } peers { permanent_uuid: "907eed367beb47bc9d8e4a30dcdc710b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44021 } }
I20260620 14:52:11.844528 27109 heartbeater.cc:503] Master 127.8.221.254:46683 was elected leader, sending a full tablet report...
I20260620 14:52:11.844944 27127 ts_tablet_manager.cc:1428] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 14:52:11.844758 27128 raft_consensus.cc:348] T ecf92d7448b44f67bf8848c4268d50d7 P ce5edaf4bbe74d88baa835539c12b542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ab904efffc468b92f4c1b91d42e726" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38383 } } peers { permanent_uuid: "ce5edaf4bbe74d88baa835539c12b542" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43017 } } peers { permanent_uuid: "907eed367beb47bc9d8e4a30dcdc710b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44021 } }
I20260620 14:52:11.845039 27128 raft_consensus.cc:374] T ecf92d7448b44f67bf8848c4268d50d7 P ce5edaf4bbe74d88baa835539c12b542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:11.845088 27128 raft_consensus.cc:729] T ecf92d7448b44f67bf8848c4268d50d7 P ce5edaf4bbe74d88baa835539c12b542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5edaf4bbe74d88baa835539c12b542, State: Initialized, Role: FOLLOWER
I20260620 14:52:11.845567 27128 consensus_queue.cc:260] T ecf92d7448b44f67bf8848c4268d50d7 P ce5edaf4bbe74d88baa835539c12b542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ab904efffc468b92f4c1b91d42e726" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38383 } } peers { permanent_uuid: "ce5edaf4bbe74d88baa835539c12b542" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43017 } } peers { permanent_uuid: "907eed367beb47bc9d8e4a30dcdc710b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44021 } }
I20260620 14:52:11.846421 26847 heartbeater.cc:503] Master 127.8.221.254:46683 was elected leader, sending a full tablet report...
I20260620 14:52:11.846673 27128 ts_tablet_manager.cc:1428] T ecf92d7448b44f67bf8848c4268d50d7 P ce5edaf4bbe74d88baa835539c12b542: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 14:52:12.015547 27110 tablet.cc:2369] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:52:12.043519 26979 tablet.cc:2369] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:52:12.069765 26848 tablet.cc:2369] T ecf92d7448b44f67bf8848c4268d50d7 P ce5edaf4bbe74d88baa835539c12b542: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:52:12.287405 27133 raft_consensus.cc:482] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:52:12.287748 27133 raft_consensus.cc:504] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ab904efffc468b92f4c1b91d42e726" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38383 } } peers { permanent_uuid: "ce5edaf4bbe74d88baa835539c12b542" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43017 } } peers { permanent_uuid: "907eed367beb47bc9d8e4a30dcdc710b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44021 } }
I20260620 14:52:12.289315 27133 leader_election.cc:288] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61ab904efffc468b92f4c1b91d42e726 (127.8.221.194:38383), ce5edaf4bbe74d88baa835539c12b542 (127.8.221.193:43017)
I20260620 14:52:12.293123 27132 raft_consensus.cc:482] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:52:12.293411 27132 raft_consensus.cc:504] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ab904efffc468b92f4c1b91d42e726" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38383 } } peers { permanent_uuid: "ce5edaf4bbe74d88baa835539c12b542" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43017 } } peers { permanent_uuid: "907eed367beb47bc9d8e4a30dcdc710b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44021 } }
I20260620 14:52:12.294665 27121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:12.289844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:46815 (local address 127.8.221.194:38383)
0620 14:52:12.290079 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:12.290088 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:12.290230 (+ 142us) server_negotiation.cc:407] Connection header received
0620 14:52:12.290617 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:12.290625 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:12.290708 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:12.290788 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:12.291196 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.291912 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.293242 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.293951 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.294087 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:12.294147 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:12.294416 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 14:52:12.294542 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 14:52:12.294891 27132 leader_election.cc:288] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce5edaf4bbe74d88baa835539c12b542 (127.8.221.193:43017), 907eed367beb47bc9d8e4a30dcdc710b (127.8.221.195:44021)
I20260620 14:52:12.295084 27135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:12.289641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:38383 (local address 127.8.221.195:46815)
0620 14:52:12.290053 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:12.290074 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:12.290289 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:12.290824 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:12.290828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:12.290854 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:12.291041 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:12.291050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.292033 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.292037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:12.293095 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:12.293106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.294092 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.294096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:12.294217 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:12.294247 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 14:52:12.294804 (+ 557us) client_negotiation.cc:240] Negotiation successful
0620 14:52:12.294899 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":6,"thread_start_us":153,"threads_started":1}
I20260620 14:52:12.295296 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:12.290372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:43721 (local address 127.8.221.193:43017)
0620 14:52:12.290716 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:12.290724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:12.290766 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:52:12.290830 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:12.290833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:12.290911 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:12.291024 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:12.291402 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.292091 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.293673 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.294500 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.294645 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:12.294702 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:12.294980 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 14:52:12.295084 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244}
I20260620 14:52:12.295771 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:12.290166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:43017 (local address 127.8.221.195:43721)
0620 14:52:12.290551 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:12.290570 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:12.290684 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:12.291044 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:12.291046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:12.291062 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:12.291238 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:12.291246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.292276 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.292279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:12.293492 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:12.293503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.294685 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.294688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:12.294816 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:12.294850 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 14:52:12.295476 (+ 626us) client_negotiation.cc:240] Negotiation successful
0620 14:52:12.295585 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":2,"thread_start_us":132,"threads_started":1}
I20260620 14:52:12.296460 26932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf92d7448b44f67bf8848c4268d50d7" candidate_uuid: "907eed367beb47bc9d8e4a30dcdc710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ab904efffc468b92f4c1b91d42e726" is_pre_election: true
I20260620 14:52:12.296667 26801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf92d7448b44f67bf8848c4268d50d7" candidate_uuid: "907eed367beb47bc9d8e4a30dcdc710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce5edaf4bbe74d88baa835539c12b542" is_pre_election: true
I20260620 14:52:12.296767 26932 raft_consensus.cc:2454] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 907eed367beb47bc9d8e4a30dcdc710b in term 0.
I20260620 14:52:12.296983 26801 raft_consensus.cc:2454] T ecf92d7448b44f67bf8848c4268d50d7 P ce5edaf4bbe74d88baa835539c12b542 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 907eed367beb47bc9d8e4a30dcdc710b in term 0.
I20260620 14:52:12.297402 26997 leader_election.cc:302] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61ab904efffc468b92f4c1b91d42e726, 907eed367beb47bc9d8e4a30dcdc710b; no voters:
I20260620 14:52:12.297755 27133 raft_consensus.cc:2781] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:52:12.297871 27133 raft_consensus.cc:482] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:52:12.297923 27133 raft_consensus.cc:3037] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:12.299454 27133 raft_consensus.cc:504] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ab904efffc468b92f4c1b91d42e726" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38383 } } peers { permanent_uuid: "ce5edaf4bbe74d88baa835539c12b542" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43017 } } peers { permanent_uuid: "907eed367beb47bc9d8e4a30dcdc710b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44021 } }
I20260620 14:52:12.299924 27133 leader_election.cc:288] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [CANDIDATE]: Term 1 election: Requested vote from peers 61ab904efffc468b92f4c1b91d42e726 (127.8.221.194:38383), ce5edaf4bbe74d88baa835539c12b542 (127.8.221.193:43017)
I20260620 14:52:12.300352 26801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf92d7448b44f67bf8848c4268d50d7" candidate_uuid: "907eed367beb47bc9d8e4a30dcdc710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce5edaf4bbe74d88baa835539c12b542"
I20260620 14:52:12.300382 26932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf92d7448b44f67bf8848c4268d50d7" candidate_uuid: "907eed367beb47bc9d8e4a30dcdc710b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61ab904efffc468b92f4c1b91d42e726"
I20260620 14:52:12.300511 26932 raft_consensus.cc:3037] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:12.300505 26801 raft_consensus.cc:3037] T ecf92d7448b44f67bf8848c4268d50d7 P ce5edaf4bbe74d88baa835539c12b542 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:12.301025 27123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:12.295528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:57533 (local address 127.8.221.195:44021)
0620 14:52:12.295635 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:12.295640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:12.296152 (+ 512us) server_negotiation.cc:407] Connection header received
0620 14:52:12.296391 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:12.296397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:12.296444 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:12.296514 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:12.297089 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.297850 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.299666 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.300248 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.300333 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:12.300379 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:12.300787 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 14:52:12.300862 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:52:12.301193 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:12.295778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:37585 (local address 127.8.221.193:43017)
0620 14:52:12.295959 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:12.295964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:12.296241 (+ 277us) server_negotiation.cc:407] Connection header received
0620 14:52:12.296438 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:12.296441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:12.296492 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:12.296538 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:12.297153 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.297924 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.299666 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.300248 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.300333 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:12.300379 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:12.300990 (+ 611us) server_negotiation.cc:299] Negotiation successful
0620 14:52:12.301069 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 14:52:12.301386 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:12.295224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:44021 (local address 127.8.221.194:57533)
0620 14:52:12.296058 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:12.296076 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:12.296242 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:12.296739 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:12.296742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:12.296754 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:12.296970 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:12.296979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.298024 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.298028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:12.299524 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:12.299535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.300476 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.300480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:12.300641 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:12.300667 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:52:12.301145 (+ 478us) client_negotiation.cc:240] Negotiation successful
0620 14:52:12.301230 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":184,"thread_start_us":89,"threads_started":1}
I20260620 14:52:12.301630 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:12.295725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:43017 (local address 127.8.221.194:37585)
0620 14:52:12.296160 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:12.296179 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:12.296316 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:12.296670 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:12.296674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:12.296693 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:12.296937 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:12.296946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.298039 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.298042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:12.299524 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:12.299535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.300476 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.300480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:12.300858 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:12.300880 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:52:12.301416 (+ 536us) client_negotiation.cc:240] Negotiation successful
0620 14:52:12.301504 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":177,"thread_start_us":105,"threads_started":1}
I20260620 14:52:12.302111 26932 raft_consensus.cc:2454] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 907eed367beb47bc9d8e4a30dcdc710b in term 1.
I20260620 14:52:12.302111 26801 raft_consensus.cc:2454] T ecf92d7448b44f67bf8848c4268d50d7 P ce5edaf4bbe74d88baa835539c12b542 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 907eed367beb47bc9d8e4a30dcdc710b in term 1.
I20260620 14:52:12.302412 27063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf92d7448b44f67bf8848c4268d50d7" candidate_uuid: "61ab904efffc468b92f4c1b91d42e726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "907eed367beb47bc9d8e4a30dcdc710b" is_pre_election: true
I20260620 14:52:12.302584 26997 leader_election.cc:302] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 907eed367beb47bc9d8e4a30dcdc710b, ce5edaf4bbe74d88baa835539c12b542; no voters:
I20260620 14:52:12.302701 27063 raft_consensus.cc:2379] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61ab904efffc468b92f4c1b91d42e726 in current term 1: Already voted for candidate 907eed367beb47bc9d8e4a30dcdc710b in this term.
I20260620 14:52:12.302629 26801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf92d7448b44f67bf8848c4268d50d7" candidate_uuid: "61ab904efffc468b92f4c1b91d42e726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce5edaf4bbe74d88baa835539c12b542" is_pre_election: true
I20260620 14:52:12.302874 26801 raft_consensus.cc:2379] T ecf92d7448b44f67bf8848c4268d50d7 P ce5edaf4bbe74d88baa835539c12b542 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 61ab904efffc468b92f4c1b91d42e726 in current term 1: Already voted for candidate 907eed367beb47bc9d8e4a30dcdc710b in this term.
I20260620 14:52:12.302865 27133 raft_consensus.cc:2781] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:12.303191 27133 raft_consensus.cc:686] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [term 1 LEADER]: Becoming Leader. State: Replica: 907eed367beb47bc9d8e4a30dcdc710b, State: Running, Role: LEADER
I20260620 14:52:12.303505 26866 leader_election.cc:302] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61ab904efffc468b92f4c1b91d42e726; no voters: 907eed367beb47bc9d8e4a30dcdc710b, ce5edaf4bbe74d88baa835539c12b542
I20260620 14:52:12.303725 27133 consensus_queue.cc:237] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ab904efffc468b92f4c1b91d42e726" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38383 } } peers { permanent_uuid: "ce5edaf4bbe74d88baa835539c12b542" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43017 } } peers { permanent_uuid: "907eed367beb47bc9d8e4a30dcdc710b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44021 } }
I20260620 14:52:12.303920 27132 raft_consensus.cc:2726] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 14:52:12.307657 26659 catalog_manager.cc:5495] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b reported cstate change: term changed from 0 to 1, leader changed from <none> to 907eed367beb47bc9d8e4a30dcdc710b (127.8.221.195). New cstate: current_term: 1 leader_uuid: "907eed367beb47bc9d8e4a30dcdc710b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61ab904efffc468b92f4c1b91d42e726" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce5edaf4bbe74d88baa835539c12b542" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "907eed367beb47bc9d8e4a30dcdc710b" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44021 } health_report { overall_health: HEALTHY } } }
I20260620 14:52:12.386030 27123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:12.381465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46732 (local address 127.8.221.195:44021)
0620 14:52:12.381667 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:12.381681 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:12.381825 (+ 144us) server_negotiation.cc:407] Connection header received
0620 14:52:12.382064 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:12.382066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:12.382132 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:12.382265 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:12.382553 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.383132 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.384224 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.384459 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.384576 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:12.384943 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:12.385563 (+ 620us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:12.385724 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:12.385759 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 14:52:12.385851 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 14:52:12.393388 26801 raft_consensus.cc:1261] T ecf92d7448b44f67bf8848c4268d50d7 P ce5edaf4bbe74d88baa835539c12b542 [term 1 FOLLOWER]: Refusing update from remote peer 907eed367beb47bc9d8e4a30dcdc710b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:12.393503 26932 raft_consensus.cc:1261] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726 [term 1 FOLLOWER]: Refusing update from remote peer 907eed367beb47bc9d8e4a30dcdc710b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:12.394085 27133 consensus_queue.cc:1035] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce5edaf4bbe74d88baa835539c12b542" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 43017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:12.394358 27139 consensus_queue.cc:1035] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [LEADER]: Connected to new peer: Peer: permanent_uuid: "61ab904efffc468b92f4c1b91d42e726" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 38383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:12.401566 27146 mvcc.cc:205] Tried to move back new op lower bound from 7298937374274084864 to 7298937373919760384. Current Snapshot: MvccSnapshot[applied={T|T < 7298937374274084864}]
I20260620 14:52:12.402983 27148 mvcc.cc:205] Tried to move back new op lower bound from 7298937374274084864 to 7298937373919760384. Current Snapshot: MvccSnapshot[applied={T|T < 7298937374274084864}]
I20260620 14:52:12.412364 26657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:12.412632 26657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:12.415552 26657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66ee9253e44e4272b6280aa673deab0e: Sending DeleteTablet for 3 replicas of tablet ecf92d7448b44f67bf8848c4268d50d7
I20260620 14:52:12.416563 26912 tablet_service.cc:1514] Processing DeleteTablet for tablet ecf92d7448b44f67bf8848c4268d50d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:12 UTC) from {username='slave'} at 127.0.0.1:39408
I20260620 14:52:12.416595 27043 tablet_service.cc:1514] Processing DeleteTablet for tablet ecf92d7448b44f67bf8848c4268d50d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:12 UTC) from {username='slave'} at 127.0.0.1:46720
I20260620 14:52:12.416656 26781 tablet_service.cc:1514] Processing DeleteTablet for tablet ecf92d7448b44f67bf8848c4268d50d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:12 UTC) from {username='slave'} at 127.0.0.1:45488
I20260620 14:52:12.417258 27156 tablet_replica.cc:330] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726: stopping tablet replica
I20260620 14:52:12.417374 27158 tablet_replica.cc:330] T ecf92d7448b44f67bf8848c4268d50d7 P ce5edaf4bbe74d88baa835539c12b542: stopping tablet replica
I20260620 14:52:12.417649 27156 raft_consensus.cc:2229] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:12.417737 27158 raft_consensus.cc:2229] T ecf92d7448b44f67bf8848c4268d50d7 P ce5edaf4bbe74d88baa835539c12b542 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:12.418043 27156 raft_consensus.cc:2258] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:12.418123 27158 raft_consensus.cc:2258] T ecf92d7448b44f67bf8848c4268d50d7 P ce5edaf4bbe74d88baa835539c12b542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:12.418212 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 26719
I20260620 14:52:12.418512 27157 tablet_replica.cc:330] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b: stopping tablet replica
I20260620 14:52:12.418768 27157 raft_consensus.cc:2229] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:52:12.419173 27157 raft_consensus.cc:2258] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:12.421000 27156 ts_tablet_manager.cc:1905] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:12.421902 27157 ts_tablet_manager.cc:1905] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:12.424690 27156 ts_tablet_manager.cc:1918] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:12.424832 27156 log.cc:1196] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/ecf92d7448b44f67bf8848c4268d50d7
I20260620 14:52:12.425165 27157 ts_tablet_manager.cc:1918] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:12.425227 27156 ts_tablet_manager.cc:1939] T ecf92d7448b44f67bf8848c4268d50d7 P 61ab904efffc468b92f4c1b91d42e726: Deleting consensus metadata
I20260620 14:52:12.425264 27157 log.cc:1196] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/ecf92d7448b44f67bf8848c4268d50d7
I20260620 14:52:12.425572 27157 ts_tablet_manager.cc:1939] T ecf92d7448b44f67bf8848c4268d50d7 P 907eed367beb47bc9d8e4a30dcdc710b: Deleting consensus metadata
I20260620 14:52:12.426976 26645 catalog_manager.cc:4841] TS 907eed367beb47bc9d8e4a30dcdc710b (127.8.221.195:44021): tablet ecf92d7448b44f67bf8848c4268d50d7 (table test-table [id=02cac29d69f049989e916e6f6d7cbbc8]) successfully deleted
I20260620 14:52:12.427009 26644 catalog_manager.cc:4841] TS 61ab904efffc468b92f4c1b91d42e726 (127.8.221.194:38383): tablet ecf92d7448b44f67bf8848c4268d50d7 (table test-table [id=02cac29d69f049989e916e6f6d7cbbc8]) successfully deleted
W20260620 14:52:12.428261 26644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.221.193:43017 (error 108)
I20260620 14:52:12.429215 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 26850
I20260620 14:52:12.430507 27160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:12.429165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:43017 (local address 127.0.0.1:45498)
0620 14:52:12.430246 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:12.430335 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.193:43017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":977,"thread_start_us":76,"threads_started":1}
W20260620 14:52:12.430771 26644 catalog_manager.cc:4584] TS ce5edaf4bbe74d88baa835539c12b542 (127.8.221.193:43017): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ecf92d7448b44f67bf8848c4268d50d7: Network error: Client connection negotiation failed: client connection to 127.8.221.193:43017: connect: Connection refused (error 111)
I20260620 14:52:12.439420 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 26981
I20260620 14:52:12.448113 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 26628
2026-06-20T14:52:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1708 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:12 dist-test-slave-65tz krb5kdc[27163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:12 dist-test-slave-65tz krb5kdc[27163](info): set up 2 sockets
Jun 20 14:52:12 dist-test-slave-65tz krb5kdc[27163](info): commencing operation
krb5kdc: starting...
W20260620 14:52:13.554219 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:13 dist-test-slave-65tz krb5kdc[27163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:52:13.685143 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:33711
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:33711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:13.815003 27176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:13.815335 27176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:13.815382 27176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:13.819625 27176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:13.819694 27176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:13.819715 27176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:13.819736 27176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:13.819752 27176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:13.825531 27176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:33711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:33711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:13.827101 27176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:13.828429 27176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:13.838898 27182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:13.838901 27184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:13.838892 27181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:13.839406 27176 server_base.cc:1034] running on GCE node
I20260620 14:52:13.840288 27176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:13.841759 27176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:13.842998 27176 hybrid_clock.cc:648] HybridClock initialized: now 1781967133842978 us; error 37 us; skew 500 ppm
Jun 20 14:52:13 dist-test-slave-65tz krb5kdc[27163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967133, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:13.850415 27176 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:52:13.851583 27176 webserver.cc:460] Webserver started at http://127.8.221.254:41359/ using document root <none> and password file <none>
I20260620 14:52:13.852262 27176 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:13.852326 27176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:13.852540 27176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:13.855224 27176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "381d76fdef874ed6a7d1975e521065d1"
format_stamp: "Formatted at 2026-06-20 14:52:13 on dist-test-slave-65tz"
server_key: "7012c773ca21ffd4396c61e9fe844055"
server_key_iv: "0d1a7e5b2d444d84373af4c9187937b1"
server_key_version: "encryptionkey@0"
I20260620 14:52:13.855909 27176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "381d76fdef874ed6a7d1975e521065d1"
format_stamp: "Formatted at 2026-06-20 14:52:13 on dist-test-slave-65tz"
server_key: "7012c773ca21ffd4396c61e9fe844055"
server_key_iv: "0d1a7e5b2d444d84373af4c9187937b1"
server_key_version: "encryptionkey@0"
I20260620 14:52:13.861039 27176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:52:13.864535 27191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:13.866564 27176 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:52:13.866775 27176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "381d76fdef874ed6a7d1975e521065d1"
format_stamp: "Formatted at 2026-06-20 14:52:13 on dist-test-slave-65tz"
server_key: "7012c773ca21ffd4396c61e9fe844055"
server_key_iv: "0d1a7e5b2d444d84373af4c9187937b1"
server_key_version: "encryptionkey@0"
I20260620 14:52:13.866952 27176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:13.882325 27176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:13.886001 27176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:13.886242 27176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:13.895339 27176 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:33711
I20260620 14:52:13.895373 27243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:33711 every 8 connection(s)
I20260620 14:52:13.896727 27176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:13.899858 27244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:13.903012 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 27176
I20260620 14:52:13.903182 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:13.903483 9079 external_mini_cluster.cc:1428] Setting key 5a38ed59e00bd5fe13464bc3d4ae6a7f
I20260620 14:52:13.906395 27244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1: Bootstrap starting.
I20260620 14:52:13.908828 27244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:13.909674 27244 log.cc:824] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:13.912084 27244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1: No bootstrap required, opened a new log
Jun 20 14:52:13 dist-test-slave-65tz krb5kdc[27163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:52:13.917196 27244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381d76fdef874ed6a7d1975e521065d1" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33711 } }
I20260620 14:52:13.917522 27244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:13.917582 27244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 381d76fdef874ed6a7d1975e521065d1, State: Initialized, Role: FOLLOWER
I20260620 14:52:13.918131 27244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381d76fdef874ed6a7d1975e521065d1" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33711 } }
I20260620 14:52:13.918309 27244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:13.918392 27244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:13.918510 27244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:13.919831 27244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381d76fdef874ed6a7d1975e521065d1" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33711 } }
I20260620 14:52:13.920217 27244 leader_election.cc:302] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 381d76fdef874ed6a7d1975e521065d1; no voters:
I20260620 14:52:13.920660 27244 leader_election.cc:288] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:13.920894 27249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:13.921218 27249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1 [term 1 LEADER]: Becoming Leader. State: Replica: 381d76fdef874ed6a7d1975e521065d1, State: Running, Role: LEADER
I20260620 14:52:13.921834 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:13.905298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.8.221.254:33711)
0620 14:52:13.906030 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:13.906060 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:13.906117 (+ 57us) server_negotiation.cc:407] Connection header received
0620 14:52:13.907133 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:13.907160 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:13.907689 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:13.908146 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:13.909199 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.910662 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.911649 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.912238 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.912416 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:13.915836 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:13.915845 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:13.915867 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:13.915909 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:13.918115 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:13.918809 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:13.918815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:13.918821 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:13.918961 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:13.919243 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:13.919246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:13.919248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:13.919829 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:13.920031 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:13.920455 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 14:52:13.920801 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":185,"threads_started":1}
I20260620 14:52:13.921759 27249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381d76fdef874ed6a7d1975e521065d1" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33711 } }
I20260620 14:52:13.922175 27244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:13.923452 27249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 381d76fdef874ed6a7d1975e521065d1. Latest consensus state: current_term: 1 leader_uuid: "381d76fdef874ed6a7d1975e521065d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381d76fdef874ed6a7d1975e521065d1" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33711 } } }
I20260620 14:52:13.923593 27249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:13.924371 27250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "381d76fdef874ed6a7d1975e521065d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "381d76fdef874ed6a7d1975e521065d1" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33711 } } }
I20260620 14:52:13.924489 27250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:13.924849 27258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:13.928432 27258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:13.935655 27258 catalog_manager.cc:1347] Generated new cluster ID: d9d86f87b1a24ff48b25c2d14eac467e
I20260620 14:52:13.935771 27258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:13.956136 27258 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:13.957700 27258 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:13.970376 27258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1: Generated new TSK 0
I20260620 14:52:13.971306 27258 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
I20260620 14:52:14.009696 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33711
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:14.146103 27270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:14.146503 27270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:14.146560 27270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:14.151211 27270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:14.151283 27270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:14.151352 27270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:52:14.157160 27270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:14.158530 27270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:14.159740 27270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:14.171195 27278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:14.171195 27276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:14.171255 27275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:14.172123 27270 server_base.cc:1034] running on GCE node
I20260620 14:52:14.172626 27270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:14.173385 27270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:14.174556 27270 hybrid_clock.cc:648] HybridClock initialized: now 1781967134174537 us; error 45 us; skew 500 ppm
Jun 20 14:52:14 dist-test-slave-65tz krb5kdc[27163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967134, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:14.177914 27270 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:52:14.179018 27270 webserver.cc:460] Webserver started at http://127.8.221.193:34669/ using document root <none> and password file <none>
I20260620 14:52:14.179669 27270 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:14.179731 27270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:14.179939 27270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:14.182375 27270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "d126007eba3040fcbbf3059f107c3bcd"
format_stamp: "Formatted at 2026-06-20 14:52:14 on dist-test-slave-65tz"
server_key: "adc0967eed0938981963af9c89e1b21f"
server_key_iv: "0272ae71528559d5ab31d476c01be481"
server_key_version: "encryptionkey@0"
I20260620 14:52:14.182974 27270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "d126007eba3040fcbbf3059f107c3bcd"
format_stamp: "Formatted at 2026-06-20 14:52:14 on dist-test-slave-65tz"
server_key: "adc0967eed0938981963af9c89e1b21f"
server_key_iv: "0272ae71528559d5ab31d476c01be481"
server_key_version: "encryptionkey@0"
I20260620 14:52:14.187776 27270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:52:14.190969 27285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:14.192822 27270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:52:14.192960 27270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "d126007eba3040fcbbf3059f107c3bcd"
format_stamp: "Formatted at 2026-06-20 14:52:14 on dist-test-slave-65tz"
server_key: "adc0967eed0938981963af9c89e1b21f"
server_key_iv: "0272ae71528559d5ab31d476c01be481"
server_key_version: "encryptionkey@0"
I20260620 14:52:14.193084 27270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:14.220294 27270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:14.224606 27270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:14.224807 27270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:14.225534 27270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:14.226621 27270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:14.226678 27270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:14.226727 27270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:14.226742 27270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:14.239040 27270 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:45853
I20260620 14:52:14.239123 27398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:45853 every 8 connection(s)
I20260620 14:52:14.240201 27270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:52:14.248749 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 27270
I20260620 14:52:14.248940 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:52:14.249365 9079 external_mini_cluster.cc:1428] Setting key 87eabc54c72312b2334985b6a3cb9835
Jun 20 14:52:14 dist-test-slave-65tz krb5kdc[27163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967134, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:52:14.255713 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.242559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:39675 (local address 127.8.221.254:33711)
0620 14:52:14.242777 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:14.242781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:14.243789 (+ 1008us) server_negotiation.cc:407] Connection header received
0620 14:52:14.244926 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:14.244929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:14.244983 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:14.245073 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:14.246326 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.246869 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.248014 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.248243 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.248352 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.251591 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:14.251596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:14.251599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:14.251623 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:14.253300 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:14.253959 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:14.253962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:14.253964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:14.254061 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:14.254445 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:14.254448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:14.254450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:14.254665 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:14.254781 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:14.255369 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 14:52:14.255527 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 14:52:14.256840 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.243017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:33711 (local address 127.8.221.193:39675)
0620 14:52:14.243599 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:14.243651 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:14.244708 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:14.245270 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:14.245277 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:14.245659 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:14.246180 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.246193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.246996 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.247004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.247880 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.247888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.248398 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.248402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.248577 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.249299 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:14.249327 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:14.251376 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:14.253478 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:14.253484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:14.253503 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:14.253845 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:14.254160 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:14.254165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:14.254169 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:14.254334 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:14.254778 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:14.254786 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:14.255100 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 14:52:14.255316 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 14:52:14.255718 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":129,"threads_started":1}
I20260620 14:52:14.258548 27399 heartbeater.cc:347] Connected to a master server at 127.8.221.254:33711
I20260620 14:52:14.258944 27399 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.221.194@KRBTEST.COM; defaulting to no policy
I20260620 14:52:14.259730 27399 heartbeater.cc:511] Master 127.8.221.254:33711 requested a full tablet report, sending...
I20260620 14:52:14.261548 27208 ts_manager.cc:194] Registered new tserver with Master: d126007eba3040fcbbf3059f107c3bcd (127.8.221.193:45853)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.194@KRBTEST.COM" created.
I20260620 14:52:14.263213 27208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:39675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
I20260620 14:52:14.280750 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33711
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:14.408389 27404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:14.408756 27404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:14.408874 27404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:14.413506 27404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:14.413591 27404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:14.413657 27404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:52:14.419895 27404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:14.421402 27404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:14.422835 27404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:14.435282 27409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:14.435282 27410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:14.435303 27412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:14.435793 27404 server_base.cc:1034] running on GCE node
I20260620 14:52:14.436313 27404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:14.437095 27404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:14.438293 27404 hybrid_clock.cc:648] HybridClock initialized: now 1781967134438246 us; error 63 us; skew 500 ppm
Jun 20 14:52:14 dist-test-slave-65tz krb5kdc[27163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967134, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:14.441519 27404 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:52:14.442632 27404 webserver.cc:460] Webserver started at http://127.8.221.194:41607/ using document root <none> and password file <none>
I20260620 14:52:14.443367 27404 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:14.443464 27404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:14.443738 27404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:14.446342 27404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "9c0900ed046244fdae0945b600a63423"
format_stamp: "Formatted at 2026-06-20 14:52:14 on dist-test-slave-65tz"
server_key: "ec55ec911684280feb76de5733fe57b8"
server_key_iv: "29caefb954cf945f4b81b227d76eab09"
server_key_version: "encryptionkey@0"
I20260620 14:52:14.446992 27404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "9c0900ed046244fdae0945b600a63423"
format_stamp: "Formatted at 2026-06-20 14:52:14 on dist-test-slave-65tz"
server_key: "ec55ec911684280feb76de5733fe57b8"
server_key_iv: "29caefb954cf945f4b81b227d76eab09"
server_key_version: "encryptionkey@0"
I20260620 14:52:14.452786 27404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:52:14.455899 27419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:14.457489 27404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:52:14.457676 27404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "9c0900ed046244fdae0945b600a63423"
format_stamp: "Formatted at 2026-06-20 14:52:14 on dist-test-slave-65tz"
server_key: "ec55ec911684280feb76de5733fe57b8"
server_key_iv: "29caefb954cf945f4b81b227d76eab09"
server_key_version: "encryptionkey@0"
I20260620 14:52:14.457875 27404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:14.475304 27404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:14.479175 27404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:14.479413 27404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:14.480172 27404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:14.481145 27404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:14.481218 27404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:14.481293 27404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:14.481344 27404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:14.492949 27404 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:34683
I20260620 14:52:14.493002 27532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:34683 every 8 connection(s)
I20260620 14:52:14.494146 27404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:52:14.498741 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 27404
I20260620 14:52:14.498898 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:52:14.499255 9079 external_mini_cluster.cc:1428] Setting key c67fc6bb3cae0225c15cf47d19d47d92
Jun 20 14:52:14 dist-test-slave-65tz krb5kdc[27163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967134, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.195@KRBTEST.COM; defaulting to no policy
I20260620 14:52:14.510900 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.496302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:46389 (local address 127.8.221.254:33711)
0620 14:52:14.496526 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:14.496532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:14.497609 (+ 1077us) server_negotiation.cc:407] Connection header received
0620 14:52:14.498736 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:14.498740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:14.498821 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:14.498947 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:14.500393 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.501099 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.502320 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.502749 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.502891 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.506312 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:14.506319 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:14.506324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:14.506364 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:14.508223 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:14.508912 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:14.508917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:14.508920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:14.509018 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:14.509402 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:14.509406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:14.509408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:14.509687 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:14.509851 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:14.510521 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 14:52:14.510675 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 14:52:14.512038 27535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.496825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:33711 (local address 127.8.221.194:46389)
0620 14:52:14.497439 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:14.497489 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:14.498492 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:14.499157 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:14.499163 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:14.499622 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:14.500253 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.500266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.501247 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.501250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.502194 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.502205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.502920 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.502923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.503068 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.503876 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:14.503900 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:14.505998 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:14.508380 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:14.508386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:14.508405 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:14.508765 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:14.509128 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:14.509130 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:14.509131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:14.509279 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:14.509884 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:14.509891 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:14.510217 (+ 326us) client_negotiation.cc:769] Sending connection context
0620 14:52:14.510396 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 14:52:14.510808 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":133,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.195@KRBTEST.COM" created.
I20260620 14:52:14.513833 27533 heartbeater.cc:347] Connected to a master server at 127.8.221.254:33711
I20260620 14:52:14.514185 27533 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:14.514885 27533 heartbeater.cc:511] Master 127.8.221.254:33711 requested a full tablet report, sending...
I20260620 14:52:14.516340 27208 ts_manager.cc:194] Registered new tserver with Master: 9c0900ed046244fdae0945b600a63423 (127.8.221.194:34683)
I20260620 14:52:14.517128 27208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:46389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
I20260620 14:52:14.529557 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33711
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:14.648849 27538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:14.649147 27538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:14.649191 27538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:14.653541 27538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:14.653622 27538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:14.653723 27538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:52:14.659413 27538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:14.660584 27538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:14.661702 27538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:14.672585 27543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:14.672585 27544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:14.672565 27546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:14.673192 27538 server_base.cc:1034] running on GCE node
I20260620 14:52:14.673664 27538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:14.674513 27538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:14.675743 27538 hybrid_clock.cc:648] HybridClock initialized: now 1781967134675716 us; error 42 us; skew 500 ppm
Jun 20 14:52:14 dist-test-slave-65tz krb5kdc[27163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967134, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:14.679183 27538 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:52:14.680343 27538 webserver.cc:460] Webserver started at http://127.8.221.195:46109/ using document root <none> and password file <none>
I20260620 14:52:14.681071 27538 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:14.681166 27538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:14.681447 27538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:14.684106 27538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "48d683957db741f488933824c2ef800a"
format_stamp: "Formatted at 2026-06-20 14:52:14 on dist-test-slave-65tz"
server_key: "6a07dbcad5dfdadf0b96135ab850cc6c"
server_key_iv: "0e851c50e94655bb6b25b9137d01fe62"
server_key_version: "encryptionkey@0"
I20260620 14:52:14.684762 27538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "48d683957db741f488933824c2ef800a"
format_stamp: "Formatted at 2026-06-20 14:52:14 on dist-test-slave-65tz"
server_key: "6a07dbcad5dfdadf0b96135ab850cc6c"
server_key_iv: "0e851c50e94655bb6b25b9137d01fe62"
server_key_version: "encryptionkey@0"
I20260620 14:52:14.689481 27538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260620 14:52:14.692937 27553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:14.694800 27538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:14.694976 27538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "48d683957db741f488933824c2ef800a"
format_stamp: "Formatted at 2026-06-20 14:52:14 on dist-test-slave-65tz"
server_key: "6a07dbcad5dfdadf0b96135ab850cc6c"
server_key_iv: "0e851c50e94655bb6b25b9137d01fe62"
server_key_version: "encryptionkey@0"
I20260620 14:52:14.695127 27538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:14.708771 27538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:14.713853 27538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:14.714130 27538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:14.714949 27538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:14.715987 27538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:14.716039 27538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:14.716094 27538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:14.716110 27538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:14.727463 27538 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:39499
I20260620 14:52:14.727512 27666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:39499 every 8 connection(s)
I20260620 14:52:14.728667 27538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:52:14.738123 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 27538
I20260620 14:52:14.738286 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:52:14.738600 9079 external_mini_cluster.cc:1428] Setting key 402df1e0fff5f0f521bc3970927ae646
Jun 20 14:52:14 dist-test-slave-65tz krb5kdc[27163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967134, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:52:14.745133 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.730965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:47997 (local address 127.8.221.254:33711)
0620 14:52:14.731166 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:14.731171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:14.732320 (+ 1149us) server_negotiation.cc:407] Connection header received
0620 14:52:14.733560 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:14.733564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:14.733634 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:14.733751 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:14.735220 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.735865 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.737184 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.737429 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.737548 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.740951 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:14.740955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:14.740960 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:14.740990 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:14.742740 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:14.743341 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:14.743345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:14.743347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:14.743451 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:14.743805 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:14.743808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:14.743809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:14.744049 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:14.744142 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:14.744760 (+ 618us) server_negotiation.cc:299] Negotiation successful
0620 14:52:14.744912 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:52:14.745982 27669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.731320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:33711 (local address 127.8.221.195:47997)
0620 14:52:14.732124 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:14.732177 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:14.733287 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:14.733990 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:14.733999 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:14.734460 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:14.735061 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.735077 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.736015 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.736024 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.737048 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.737058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.737627 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.737632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.737880 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.738730 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:14.738764 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:14.740735 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:14.742883 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:14.742890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:14.742905 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:14.743221 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:14.743554 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:14.743556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:14.743557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:14.743700 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:14.744164 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:14.744170 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:14.744479 (+ 309us) client_negotiation.cc:769] Sending connection context
0620 14:52:14.744664 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 14:52:14.744974 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":148,"threads_started":1}
I20260620 14:52:14.747298 27667 heartbeater.cc:347] Connected to a master server at 127.8.221.254:33711
I20260620 14:52:14.747716 27667 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:14.748413 27667 heartbeater.cc:511] Master 127.8.221.254:33711 requested a full tablet report, sending...
I20260620 14:52:14.749902 27208 ts_manager.cc:194] Registered new tserver with Master: 48d683957db741f488933824c2ef800a (127.8.221.195:39499)
I20260620 14:52:14.750624 27208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:47997
I20260620 14:52:14.753333 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:52:14.764567 27247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.755798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53526 (local address 127.8.221.254:33711)
0620 14:52:14.755991 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:14.755997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:14.756124 (+ 127us) server_negotiation.cc:407] Connection header received
0620 14:52:14.756237 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:14.756243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:14.756314 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:14.756447 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:14.757224 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.758205 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.759046 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.759264 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.759356 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.760941 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:14.760944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:14.760947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:14.760993 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:14.762555 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:14.763098 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:14.763101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:14.763103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:14.763191 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:14.763626 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:14.763630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:14.763631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:14.763856 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:14.764012 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:14.764305 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 14:52:14.764410 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:52:14.767987 27208 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:52:14.771915 27208 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:52:14.789754 27678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.782130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:45853 (local address 127.0.0.1:47210)
0620 14:52:14.783223 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:14.783243 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:14.783367 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:14.784300 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:14.784308 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:14.784337 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:14.784558 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.784567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.786262 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.786265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.787902 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.787914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.789175 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.789179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.789328 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.789382 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 14:52:14.789469 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:52:14.789572 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"mutex_wait_us":29,"thread_start_us":156,"threads_started":1}
I20260620 14:52:14.789971 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.782964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47210 (local address 127.8.221.193:45853)
0620 14:52:14.783661 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:14.783670 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:14.783698 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:52:14.783806 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:14.783812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:14.784089 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:14.784727 (+ 638us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:14.784882 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.786107 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.788058 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.789025 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.789283 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.789487 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:14.789556 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:52:14.789705 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":225,"threads_started":1}
I20260620 14:52:14.790239 27679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.781706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:39499 (local address 127.0.0.1:38970)
0620 14:52:14.782625 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:14.782675 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:14.782808 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:14.784576 (+ 1768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:14.784610 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:14.784632 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:14.784831 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.784862 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.787032 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.787037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.788532 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.788542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.789731 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.789735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.789871 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.789903 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 14:52:14.789989 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:52:14.790075 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":121,"thread_start_us":117,"threads_started":1}
I20260620 14:52:14.790823 27681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.783073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38970 (local address 127.8.221.195:39499)
0620 14:52:14.783721 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:14.783736 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:14.783758 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:52:14.783829 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:14.783833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:14.784089 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:14.784276 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:14.785072 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.786835 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.788717 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.789586 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.789704 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.789870 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:14.790516 (+ 646us) server_negotiation.cc:299] Negotiation successful
0620 14:52:14.790670 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":126,"threads_started":1}
I20260620 14:52:14.791049 27680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.782463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:34683 (local address 127.0.0.1:58752)
0620 14:52:14.784178 (+ 1715us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:14.784194 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:14.784295 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:14.785186 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:14.785188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:14.785207 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:14.785281 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.785290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.787466 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.787472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.788296 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.788304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.789143 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.789147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.790687 (+ 1540us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.790717 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 14:52:14.790793 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:52:14.790889 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1653,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260620 14:52:14.791607 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.783468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58752 (local address 127.8.221.194:34683)
0620 14:52:14.783961 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:14.783973 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:14.784393 (+ 420us) server_negotiation.cc:407] Connection header received
0620 14:52:14.784640 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:14.784646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:14.784880 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:14.785030 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:14.785408 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.787324 (+ 1916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.788424 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.789016 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.789144 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.789310 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:14.791282 (+ 1972us) server_negotiation.cc:299] Negotiation successful
0620 14:52:14.791449 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":155,"threads_started":1}
I20260620 14:52:14.793664 27601 tablet_service.cc:1467] Processing CreateTablet for tablet aacf3e583332414f887a2bcbb00641a8 (DEFAULT_TABLE table=test-table [id=fa6c825e7c614934bc128df7969daf27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:14.793679 27333 tablet_service.cc:1467] Processing CreateTablet for tablet aacf3e583332414f887a2bcbb00641a8 (DEFAULT_TABLE table=test-table [id=fa6c825e7c614934bc128df7969daf27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:14.794814 27601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aacf3e583332414f887a2bcbb00641a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:14.795032 27333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aacf3e583332414f887a2bcbb00641a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:14.797024 27467 tablet_service.cc:1467] Processing CreateTablet for tablet aacf3e583332414f887a2bcbb00641a8 (DEFAULT_TABLE table=test-table [id=fa6c825e7c614934bc128df7969daf27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:14.798233 27467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aacf3e583332414f887a2bcbb00641a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:14.803532 27684 tablet_bootstrap.cc:492] T aacf3e583332414f887a2bcbb00641a8 P d126007eba3040fcbbf3059f107c3bcd: Bootstrap starting.
I20260620 14:52:14.805032 27685 tablet_bootstrap.cc:492] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a: Bootstrap starting.
I20260620 14:52:14.805548 27686 tablet_bootstrap.cc:492] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423: Bootstrap starting.
I20260620 14:52:14.806295 27684 tablet_bootstrap.cc:654] T aacf3e583332414f887a2bcbb00641a8 P d126007eba3040fcbbf3059f107c3bcd: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:14.807318 27685 tablet_bootstrap.cc:654] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:14.807344 27684 log.cc:824] T aacf3e583332414f887a2bcbb00641a8 P d126007eba3040fcbbf3059f107c3bcd: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:14.807825 27686 tablet_bootstrap.cc:654] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:14.808590 27685 log.cc:824] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:14.809022 27686 log.cc:824] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:14.810145 27684 tablet_bootstrap.cc:492] T aacf3e583332414f887a2bcbb00641a8 P d126007eba3040fcbbf3059f107c3bcd: No bootstrap required, opened a new log
I20260620 14:52:14.810372 27684 ts_tablet_manager.cc:1397] T aacf3e583332414f887a2bcbb00641a8 P d126007eba3040fcbbf3059f107c3bcd: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 14:52:14.811028 27685 tablet_bootstrap.cc:492] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a: No bootstrap required, opened a new log
I20260620 14:52:14.811129 27686 tablet_bootstrap.cc:492] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423: No bootstrap required, opened a new log
I20260620 14:52:14.811251 27685 ts_tablet_manager.cc:1397] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:52:14.811312 27686 ts_tablet_manager.cc:1397] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:52:14.814222 27685 raft_consensus.cc:348] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d683957db741f488933824c2ef800a" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39499 } } peers { permanent_uuid: "d126007eba3040fcbbf3059f107c3bcd" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45853 } } peers { permanent_uuid: "9c0900ed046244fdae0945b600a63423" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34683 } }
I20260620 14:52:14.814483 27685 raft_consensus.cc:374] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:14.814553 27685 raft_consensus.cc:729] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d683957db741f488933824c2ef800a, State: Initialized, Role: FOLLOWER
I20260620 14:52:14.815021 27685 consensus_queue.cc:260] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d683957db741f488933824c2ef800a" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39499 } } peers { permanent_uuid: "d126007eba3040fcbbf3059f107c3bcd" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45853 } } peers { permanent_uuid: "9c0900ed046244fdae0945b600a63423" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34683 } }
I20260620 14:52:14.815500 27686 raft_consensus.cc:348] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d683957db741f488933824c2ef800a" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39499 } } peers { permanent_uuid: "d126007eba3040fcbbf3059f107c3bcd" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45853 } } peers { permanent_uuid: "9c0900ed046244fdae0945b600a63423" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34683 } }
I20260620 14:52:14.815768 27667 heartbeater.cc:503] Master 127.8.221.254:33711 was elected leader, sending a full tablet report...
I20260620 14:52:14.815819 27686 raft_consensus.cc:374] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:14.815932 27686 raft_consensus.cc:729] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c0900ed046244fdae0945b600a63423, State: Initialized, Role: FOLLOWER
I20260620 14:52:14.816326 27685 ts_tablet_manager.cc:1428] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:52:14.816491 27686 consensus_queue.cc:260] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d683957db741f488933824c2ef800a" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39499 } } peers { permanent_uuid: "d126007eba3040fcbbf3059f107c3bcd" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45853 } } peers { permanent_uuid: "9c0900ed046244fdae0945b600a63423" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34683 } }
I20260620 14:52:14.815500 27684 raft_consensus.cc:348] T aacf3e583332414f887a2bcbb00641a8 P d126007eba3040fcbbf3059f107c3bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d683957db741f488933824c2ef800a" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39499 } } peers { permanent_uuid: "d126007eba3040fcbbf3059f107c3bcd" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45853 } } peers { permanent_uuid: "9c0900ed046244fdae0945b600a63423" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34683 } }
I20260620 14:52:14.817253 27684 raft_consensus.cc:374] T aacf3e583332414f887a2bcbb00641a8 P d126007eba3040fcbbf3059f107c3bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:14.817308 27533 heartbeater.cc:503] Master 127.8.221.254:33711 was elected leader, sending a full tablet report...
I20260620 14:52:14.817344 27684 raft_consensus.cc:729] T aacf3e583332414f887a2bcbb00641a8 P d126007eba3040fcbbf3059f107c3bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d126007eba3040fcbbf3059f107c3bcd, State: Initialized, Role: FOLLOWER
I20260620 14:52:14.817651 27686 ts_tablet_manager.cc:1428] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:52:14.817906 27684 consensus_queue.cc:260] T aacf3e583332414f887a2bcbb00641a8 P d126007eba3040fcbbf3059f107c3bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d683957db741f488933824c2ef800a" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39499 } } peers { permanent_uuid: "d126007eba3040fcbbf3059f107c3bcd" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45853 } } peers { permanent_uuid: "9c0900ed046244fdae0945b600a63423" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34683 } }
I20260620 14:52:14.818718 27399 heartbeater.cc:503] Master 127.8.221.254:33711 was elected leader, sending a full tablet report...
I20260620 14:52:14.818789 27684 ts_tablet_manager.cc:1428] T aacf3e583332414f887a2bcbb00641a8 P d126007eba3040fcbbf3059f107c3bcd: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260620 14:52:14.919075 27691 raft_consensus.cc:482] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:52:14.919327 27691 raft_consensus.cc:504] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d683957db741f488933824c2ef800a" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39499 } } peers { permanent_uuid: "d126007eba3040fcbbf3059f107c3bcd" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45853 } } peers { permanent_uuid: "9c0900ed046244fdae0945b600a63423" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34683 } }
I20260620 14:52:14.920513 27691 leader_election.cc:288] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48d683957db741f488933824c2ef800a (127.8.221.195:39499), d126007eba3040fcbbf3059f107c3bcd (127.8.221.193:45853)
I20260620 14:52:14.925206 27535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.920919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:45853 (local address 127.8.221.194:58539)
0620 14:52:14.921102 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:14.921121 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:14.921236 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:14.921592 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:14.921596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:14.921618 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:14.921787 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.921852 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.922780 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.922786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.923887 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.923896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.924681 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.924688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.924812 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.924838 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:52:14.924926 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:52:14.925038 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"spinlock_wait_cycles":43136}
I20260620 14:52:14.925249 27681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.921180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:59925 (local address 127.8.221.195:39499)
0620 14:52:14.921324 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:14.921331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:14.921636 (+ 305us) server_negotiation.cc:407] Connection header received
0620 14:52:14.921857 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:14.921862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:14.921910 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:14.922002 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:14.922366 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.922881 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.924139 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.924655 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.924781 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.924845 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:14.925069 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 14:52:14.925136 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:52:14.925251 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.921086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:58539 (local address 127.8.221.193:45853)
0620 14:52:14.921264 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:14.921270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:14.921291 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:52:14.921375 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:14.921379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:14.921456 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:14.921574 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:14.921978 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.922665 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.924015 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.924487 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.924653 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.924707 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:14.924976 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 14:52:14.925079 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:52:14.925266 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.921002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:39499 (local address 127.8.221.194:59925)
0620 14:52:14.921532 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:14.921543 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:14.921706 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:14.922015 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:14.922018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:14.922035 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:14.922244 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.922254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.923001 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.923007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.924010 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.924019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.924789 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.924801 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.924886 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.924941 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 14:52:14.925030 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:52:14.925118 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":5,"spinlock_wait_cycles":1408,"thread_start_us":184,"threads_started":1}
I20260620 14:52:14.926131 27621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf3e583332414f887a2bcbb00641a8" candidate_uuid: "9c0900ed046244fdae0945b600a63423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d683957db741f488933824c2ef800a" is_pre_election: true
I20260620 14:52:14.926133 27353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf3e583332414f887a2bcbb00641a8" candidate_uuid: "9c0900ed046244fdae0945b600a63423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d126007eba3040fcbbf3059f107c3bcd" is_pre_election: true
I20260620 14:52:14.926483 27621 raft_consensus.cc:2454] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c0900ed046244fdae0945b600a63423 in term 0.
I20260620 14:52:14.926566 27353 raft_consensus.cc:2454] T aacf3e583332414f887a2bcbb00641a8 P d126007eba3040fcbbf3059f107c3bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c0900ed046244fdae0945b600a63423 in term 0.
I20260620 14:52:14.927042 27423 leader_election.cc:302] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48d683957db741f488933824c2ef800a, 9c0900ed046244fdae0945b600a63423; no voters:
I20260620 14:52:14.927384 27691 raft_consensus.cc:2781] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:52:14.927477 27691 raft_consensus.cc:482] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:52:14.927510 27691 raft_consensus.cc:3037] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:14.928803 27691 raft_consensus.cc:504] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d683957db741f488933824c2ef800a" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39499 } } peers { permanent_uuid: "d126007eba3040fcbbf3059f107c3bcd" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45853 } } peers { permanent_uuid: "9c0900ed046244fdae0945b600a63423" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34683 } }
I20260620 14:52:14.929239 27691 leader_election.cc:288] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [CANDIDATE]: Term 1 election: Requested vote from peers 48d683957db741f488933824c2ef800a (127.8.221.195:39499), d126007eba3040fcbbf3059f107c3bcd (127.8.221.193:45853)
I20260620 14:52:14.929740 27621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf3e583332414f887a2bcbb00641a8" candidate_uuid: "9c0900ed046244fdae0945b600a63423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d683957db741f488933824c2ef800a"
I20260620 14:52:14.929749 27353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aacf3e583332414f887a2bcbb00641a8" candidate_uuid: "9c0900ed046244fdae0945b600a63423" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d126007eba3040fcbbf3059f107c3bcd"
I20260620 14:52:14.929925 27621 raft_consensus.cc:3037] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:14.929934 27353 raft_consensus.cc:3037] T aacf3e583332414f887a2bcbb00641a8 P d126007eba3040fcbbf3059f107c3bcd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:14.931372 27353 raft_consensus.cc:2454] T aacf3e583332414f887a2bcbb00641a8 P d126007eba3040fcbbf3059f107c3bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c0900ed046244fdae0945b600a63423 in term 1.
I20260620 14:52:14.931372 27621 raft_consensus.cc:2454] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c0900ed046244fdae0945b600a63423 in term 1.
I20260620 14:52:14.931730 27422 leader_election.cc:302] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 48d683957db741f488933824c2ef800a, 9c0900ed046244fdae0945b600a63423, d126007eba3040fcbbf3059f107c3bcd; no voters:
I20260620 14:52:14.932030 27691 raft_consensus.cc:2781] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:14.932374 27691 raft_consensus.cc:686] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [term 1 LEADER]: Becoming Leader. State: Replica: 9c0900ed046244fdae0945b600a63423, State: Running, Role: LEADER
I20260620 14:52:14.932929 27691 consensus_queue.cc:237] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d683957db741f488933824c2ef800a" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39499 } } peers { permanent_uuid: "d126007eba3040fcbbf3059f107c3bcd" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45853 } } peers { permanent_uuid: "9c0900ed046244fdae0945b600a63423" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34683 } }
I20260620 14:52:14.936951 27208 catalog_manager.cc:5495] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c0900ed046244fdae0945b600a63423 (127.8.221.194). New cstate: current_term: 1 leader_uuid: "9c0900ed046244fdae0945b600a63423" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48d683957db741f488933824c2ef800a" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d126007eba3040fcbbf3059f107c3bcd" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c0900ed046244fdae0945b600a63423" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 34683 } health_report { overall_health: HEALTHY } } }
W20260620 14:52:14.980052 27668 tablet.cc:2369] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:52:14.987396 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.982919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58768 (local address 127.8.221.194:34683)
0620 14:52:14.983137 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:14.983142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:14.983160 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:52:14.983258 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:14.983261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:14.983331 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:14.983511 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:14.983733 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.984282 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.985313 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.985606 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.985714 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.986244 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:14.986842 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:14.986986 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:14.987103 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 14:52:14.987192 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260620 14:52:14.991976 27400 tablet.cc:2369] T aacf3e583332414f887a2bcbb00641a8 P d126007eba3040fcbbf3059f107c3bcd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:52:14.995112 27621 raft_consensus.cc:1261] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a [term 1 FOLLOWER]: Refusing update from remote peer 9c0900ed046244fdae0945b600a63423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:14.995236 27353 raft_consensus.cc:1261] T aacf3e583332414f887a2bcbb00641a8 P d126007eba3040fcbbf3059f107c3bcd [term 1 FOLLOWER]: Refusing update from remote peer 9c0900ed046244fdae0945b600a63423: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 14:52:14.995508 27534 tablet.cc:2369] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:52:14.996094 27691 consensus_queue.cc:1035] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d683957db741f488933824c2ef800a" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 39499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:14.996320 27694 consensus_queue.cc:1035] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d126007eba3040fcbbf3059f107c3bcd" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:15.005438 27699 mvcc.cc:205] Tried to move back new op lower bound from 7298937384931024896 to 7298937384688631808. Current Snapshot: MvccSnapshot[applied={T|T < 7298937384931024896}]
I20260620 14:52:15.007617 27697 mvcc.cc:205] Tried to move back new op lower bound from 7298937384931024896 to 7298937384688631808. Current Snapshot: MvccSnapshot[applied={T|T < 7298937384931024896 or (T in {7298937384931024896})}]
I20260620 14:52:15.008081 27698 mvcc.cc:205] Tried to move back new op lower bound from 7298937384931024896 to 7298937384688631808. Current Snapshot: MvccSnapshot[applied={T|T < 7298937384931024896}]
I20260620 14:52:15.015537 27208 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:15.015781 27208 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:15.018949 27208 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 381d76fdef874ed6a7d1975e521065d1: Sending DeleteTablet for 3 replicas of tablet aacf3e583332414f887a2bcbb00641a8
I20260620 14:52:15.020371 27601 tablet_service.cc:1514] Processing DeleteTablet for tablet aacf3e583332414f887a2bcbb00641a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:15 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:38970
I20260620 14:52:15.020568 27333 tablet_service.cc:1514] Processing DeleteTablet for tablet aacf3e583332414f887a2bcbb00641a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:15 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:47210
I20260620 14:52:15.020988 27467 tablet_service.cc:1514] Processing DeleteTablet for tablet aacf3e583332414f887a2bcbb00641a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:15 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:58752
I20260620 14:52:15.021281 27707 tablet_replica.cc:330] T aacf3e583332414f887a2bcbb00641a8 P d126007eba3040fcbbf3059f107c3bcd: stopping tablet replica
I20260620 14:52:15.021318 27706 tablet_replica.cc:330] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a: stopping tablet replica
I20260620 14:52:15.021678 27707 raft_consensus.cc:2229] T aacf3e583332414f887a2bcbb00641a8 P d126007eba3040fcbbf3059f107c3bcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:15.021739 27706 raft_consensus.cc:2229] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:15.022114 27707 raft_consensus.cc:2258] T aacf3e583332414f887a2bcbb00641a8 P d126007eba3040fcbbf3059f107c3bcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:15.022162 27706 raft_consensus.cc:2258] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:15.022581 27708 tablet_replica.cc:330] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423: stopping tablet replica
I20260620 14:52:15.022631 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 27270
I20260620 14:52:15.022953 27708 raft_consensus.cc:2229] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:52:15.023547 27708 raft_consensus.cc:2258] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:15.024436 27706 ts_tablet_manager.cc:1905] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:15.025899 27708 ts_tablet_manager.cc:1905] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:15.028132 27706 ts_tablet_manager.cc:1918] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:15.028330 27706 log.cc:1196] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/aacf3e583332414f887a2bcbb00641a8
I20260620 14:52:15.028750 27706 ts_tablet_manager.cc:1939] T aacf3e583332414f887a2bcbb00641a8 P 48d683957db741f488933824c2ef800a: Deleting consensus metadata
I20260620 14:52:15.030177 27708 ts_tablet_manager.cc:1918] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:15.030646 27195 catalog_manager.cc:4841] TS 48d683957db741f488933824c2ef800a (127.8.221.195:39499): tablet aacf3e583332414f887a2bcbb00641a8 (table test-table [id=fa6c825e7c614934bc128df7969daf27]) successfully deleted
I20260620 14:52:15.030966 27708 log.cc:1196] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/aacf3e583332414f887a2bcbb00641a8
I20260620 14:52:15.031364 27708 ts_tablet_manager.cc:1939] T aacf3e583332414f887a2bcbb00641a8 P 9c0900ed046244fdae0945b600a63423: Deleting consensus metadata
W20260620 14:52:15.032371 27194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.221.193:45853 (error 108)
I20260620 14:52:15.033038 27195 catalog_manager.cc:4841] TS 9c0900ed046244fdae0945b600a63423 (127.8.221.194:34683): tablet aacf3e583332414f887a2bcbb00641a8 (table test-table [id=fa6c825e7c614934bc128df7969daf27]) successfully deleted
I20260620 14:52:15.033301 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 27404
I20260620 14:52:15.033695 27680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:15.033327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:45853 (local address 127.0.0.1:47212)
0620 14:52:15.033483 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:15.033569 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.193:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 14:52:15.033984 27194 catalog_manager.cc:4584] TS d126007eba3040fcbbf3059f107c3bcd (127.8.221.193:45853): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aacf3e583332414f887a2bcbb00641a8: Network error: Client connection negotiation failed: client connection to 127.8.221.193:45853: connect: Connection refused (error 111)
I20260620 14:52:15.040655 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 27538
I20260620 14:52:15.048079 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 27176
2026-06-20T14:52:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2601 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T14:52:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:15Z Disabled control of system clock
I20260620 14:52:15.083250 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:33607
--webserver_interface=127.8.221.254
--webserver_port=0
--builtin_ntp_servers=127.8.221.212:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:33607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:52:15.196123 27713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:15.196417 27713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:15.196455 27713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:15.200459 27713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:15.200521 27713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:15.200537 27713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:15.200554 27713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:15.200570 27713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:15.205268 27713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:33607
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:33607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:15.206383 27713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:15.207747 27713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:15.216768 27718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:15.216727 27719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:15.216727 27721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:15.217717 27713 server_base.cc:1034] running on GCE node
I20260620 14:52:15.218485 27713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:15.219919 27713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:15.221122 27713 hybrid_clock.cc:648] HybridClock initialized: now 1781967135221097 us; error 41 us; skew 500 ppm
I20260620 14:52:15.223425 27713 webserver.cc:460] Webserver started at http://127.8.221.254:46013/ using document root <none> and password file <none>
I20260620 14:52:15.224151 27713 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:15.224222 27713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:15.224510 27713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:15.227196 27713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "0895ff7ad63248d99a694d6f1ec4d97c"
format_stamp: "Formatted at 2026-06-20 14:52:15 on dist-test-slave-65tz"
server_key: "ded6812aa46ae8b01dd4bbedb4762208"
server_key_iv: "0705ab89712a4ecc669e0b8be3ed6e36"
server_key_version: "encryptionkey@0"
I20260620 14:52:15.227871 27713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "0895ff7ad63248d99a694d6f1ec4d97c"
format_stamp: "Formatted at 2026-06-20 14:52:15 on dist-test-slave-65tz"
server_key: "ded6812aa46ae8b01dd4bbedb4762208"
server_key_iv: "0705ab89712a4ecc669e0b8be3ed6e36"
server_key_version: "encryptionkey@0"
I20260620 14:52:15.232447 27713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:52:15.235539 27727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:15.237172 27713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:52:15.237388 27713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "0895ff7ad63248d99a694d6f1ec4d97c"
format_stamp: "Formatted at 2026-06-20 14:52:15 on dist-test-slave-65tz"
server_key: "ded6812aa46ae8b01dd4bbedb4762208"
server_key_iv: "0705ab89712a4ecc669e0b8be3ed6e36"
server_key_version: "encryptionkey@0"
I20260620 14:52:15.237553 27713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:15.258281 27713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:15.259248 27713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:15.259516 27713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:15.269032 27713 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:33607
I20260620 14:52:15.269037 27779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:33607 every 8 connection(s)
I20260620 14:52:15.270640 27713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:15.274570 27780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:15.279875 27780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c: Bootstrap starting.
I20260620 14:52:15.280686 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 27713
I20260620 14:52:15.280882 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:15.281244 9079 external_mini_cluster.cc:1428] Setting key f4fcab008e40c29a37fe91c79e5c0822
I20260620 14:52:15.282581 27780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:15.283433 27780 log.cc:824] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:15.285799 27780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c: No bootstrap required, opened a new log
I20260620 14:52:15.290249 27780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0895ff7ad63248d99a694d6f1ec4d97c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33607 } }
I20260620 14:52:15.290526 27780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:15.290597 27780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0895ff7ad63248d99a694d6f1ec4d97c, State: Initialized, Role: FOLLOWER
I20260620 14:52:15.291103 27780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0895ff7ad63248d99a694d6f1ec4d97c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33607 } }
I20260620 14:52:15.291260 27780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:15.291400 27780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:15.291525 27780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:15.292517 27780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0895ff7ad63248d99a694d6f1ec4d97c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33607 } }
I20260620 14:52:15.292778 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:15.282785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44756 (local address 127.8.221.254:33607)
0620 14:52:15.283686 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:15.283705 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:15.283753 (+ 48us) server_negotiation.cc:407] Connection header received
0620 14:52:15.284789 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:15.284813 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:15.285236 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:15.285609 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:15.286185 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:15.287268 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:15.288613 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:15.289151 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:15.289303 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:15.290383 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:15.290390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:15.290405 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:15.291041 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:15.291115 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:15.291139 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:15.291256 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:15.291432 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 14:52:15.291803 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":191,"threads_started":1}
I20260620 14:52:15.292907 27780 leader_election.cc:302] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0895ff7ad63248d99a694d6f1ec4d97c; no voters:
I20260620 14:52:15.293341 27780 leader_election.cc:288] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:15.293426 27785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:15.293658 27785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c [term 1 LEADER]: Becoming Leader. State: Replica: 0895ff7ad63248d99a694d6f1ec4d97c, State: Running, Role: LEADER
I20260620 14:52:15.294082 27785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0895ff7ad63248d99a694d6f1ec4d97c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33607 } }
I20260620 14:52:15.294788 27780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:15.296013 27787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0895ff7ad63248d99a694d6f1ec4d97c. Latest consensus state: current_term: 1 leader_uuid: "0895ff7ad63248d99a694d6f1ec4d97c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0895ff7ad63248d99a694d6f1ec4d97c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33607 } } }
I20260620 14:52:15.296187 27787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:15.296552 27786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0895ff7ad63248d99a694d6f1ec4d97c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0895ff7ad63248d99a694d6f1ec4d97c" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33607 } } }
I20260620 14:52:15.296667 27786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c [sys.catalog]: This master's current role is: LEADER
W20260620 14:52:15.300010 27800 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:52:15.300107 27800 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:52:15.300238 27795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:15.301259 27795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:15.308769 27795 catalog_manager.cc:1347] Generated new cluster ID: 21f1c5c14c0b4870bed53baf83acde03
I20260620 14:52:15.308893 27795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:15.331037 27795 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:15.332444 27795 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:15.337354 27795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c: Generated new TSK 0
I20260620 14:52:15.338392 27795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:15.347050 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33607
--builtin_ntp_servers=127.8.221.212:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:52:15.485476 27804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:15.485909 27804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:15.486001 27804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:15.490463 27804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:15.490586 27804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:15.490701 27804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:52:15.496260 27804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:15.497823 27804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:15.499559 27804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:15.511274 27810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:15.511283 27809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:15.511620 27804 server_base.cc:1034] running on GCE node
W20260620 14:52:15.511792 27812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:15.512316 27804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:15.513101 27804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:15.514303 27804 hybrid_clock.cc:648] HybridClock initialized: now 1781967135514273 us; error 55 us; skew 500 ppm
I20260620 14:52:15.516571 27804 webserver.cc:460] Webserver started at http://127.8.221.193:43677/ using document root <none> and password file <none>
I20260620 14:52:15.517290 27804 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:15.517393 27804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:15.517669 27804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:15.520233 27804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "10c9422e5dc14569a2860051a850326c"
format_stamp: "Formatted at 2026-06-20 14:52:15 on dist-test-slave-65tz"
server_key: "9894a4f1d46c619b804f65604e1c398a"
server_key_iv: "ed39a9716820f565c971e1a83fc1b0ae"
server_key_version: "encryptionkey@0"
I20260620 14:52:15.520905 27804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "10c9422e5dc14569a2860051a850326c"
format_stamp: "Formatted at 2026-06-20 14:52:15 on dist-test-slave-65tz"
server_key: "9894a4f1d46c619b804f65604e1c398a"
server_key_iv: "ed39a9716820f565c971e1a83fc1b0ae"
server_key_version: "encryptionkey@0"
I20260620 14:52:15.525586 27804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 14:52:15.528676 27818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:15.530328 27804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:15.530483 27804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "10c9422e5dc14569a2860051a850326c"
format_stamp: "Formatted at 2026-06-20 14:52:15 on dist-test-slave-65tz"
server_key: "9894a4f1d46c619b804f65604e1c398a"
server_key_iv: "ed39a9716820f565c971e1a83fc1b0ae"
server_key_version: "encryptionkey@0"
I20260620 14:52:15.530622 27804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:15.558332 27804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:15.559263 27804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:15.559500 27804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:15.560256 27804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:15.561414 27804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:15.561489 27804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:15.561573 27804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:15.561615 27804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:15.573247 27804 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:46241
I20260620 14:52:15.573299 27931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:46241 every 8 connection(s)
I20260620 14:52:15.574424 27804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:52:15.575778 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 27804
I20260620 14:52:15.575939 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:52:15.576323 9079 external_mini_cluster.cc:1428] Setting key b2be8edbfe464bb1aa654f4a643613a0
I20260620 14:52:15.578689 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33607
--builtin_ntp_servers=127.8.221.212:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:52:15.584437 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:15.576524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:44331 (local address 127.8.221.254:33607)
0620 14:52:15.576737 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:15.576744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:15.577991 (+ 1247us) server_negotiation.cc:407] Connection header received
0620 14:52:15.579552 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:15.579559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:15.579631 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:15.579750 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:15.580660 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:15.581228 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:15.582485 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:15.582707 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:15.582834 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:15.583933 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:15.583937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:15.583940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:15.583959 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:15.583977 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:15.583984 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:15.584062 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:15.584222 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 14:52:15.584286 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:52:15.585263 27934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:15.577033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:33607 (local address 127.8.221.193:44331)
0620 14:52:15.577837 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:15.577889 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:15.579331 (+ 1442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:15.579892 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:15.579898 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:15.579976 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:15.580473 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:15.580485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:15.581386 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:15.581389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:15.582355 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:15.582365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:15.582857 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:15.582861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:15.583028 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:15.583678 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:52:15.583680 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:15.583716 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:52:15.583775 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:15.584074 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:15.584085 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:15.584093 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:52:15.584221 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 14:52:15.584483 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":159,"threads_started":1}
I20260620 14:52:15.586730 27932 heartbeater.cc:347] Connected to a master server at 127.8.221.254:33607
I20260620 14:52:15.587026 27932 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:15.587605 27932 heartbeater.cc:511] Master 127.8.221.254:33607 requested a full tablet report, sending...
I20260620 14:52:15.589721 27744 ts_manager.cc:194] Registered new tserver with Master: 10c9422e5dc14569a2860051a850326c (127.8.221.193:46241)
I20260620 14:52:15.591707 27744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.221.193:44331
W20260620 14:52:15.707374 27935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:15.707703 27935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:15.707748 27935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:15.712203 27935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:15.712281 27935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:15.712347 27935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:52:15.717535 27935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:15.718981 27935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:15.720629 27935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:15.731791 27941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:15.731804 27940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:15.731840 27943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:15.732198 27935 server_base.cc:1034] running on GCE node
I20260620 14:52:15.732688 27935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:15.733446 27935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:15.734609 27935 hybrid_clock.cc:648] HybridClock initialized: now 1781967135734584 us; error 46 us; skew 500 ppm
I20260620 14:52:15.736742 27935 webserver.cc:460] Webserver started at http://127.8.221.194:45589/ using document root <none> and password file <none>
I20260620 14:52:15.737380 27935 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:15.737445 27935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:15.737685 27935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:15.739998 27935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "eab2a14e7843454fb05959d3d7a7fa46"
format_stamp: "Formatted at 2026-06-20 14:52:15 on dist-test-slave-65tz"
server_key: "5c968358518ad8a62352b8364e54712d"
server_key_iv: "f479a861ab3f7bab3d85c9ea0ca61ab4"
server_key_version: "encryptionkey@0"
I20260620 14:52:15.740520 27935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "eab2a14e7843454fb05959d3d7a7fa46"
format_stamp: "Formatted at 2026-06-20 14:52:15 on dist-test-slave-65tz"
server_key: "5c968358518ad8a62352b8364e54712d"
server_key_iv: "f479a861ab3f7bab3d85c9ea0ca61ab4"
server_key_version: "encryptionkey@0"
I20260620 14:52:15.744756 27935 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:52:15.747706 27949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:15.749356 27935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:52:15.749468 27935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "eab2a14e7843454fb05959d3d7a7fa46"
format_stamp: "Formatted at 2026-06-20 14:52:15 on dist-test-slave-65tz"
server_key: "5c968358518ad8a62352b8364e54712d"
server_key_iv: "f479a861ab3f7bab3d85c9ea0ca61ab4"
server_key_version: "encryptionkey@0"
I20260620 14:52:15.749563 27935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:15.772217 27935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:15.772907 27935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:15.773058 27935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:15.773604 27935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:15.774653 27935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:15.774705 27935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:15.774777 27935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:15.774835 27935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:15.785441 27935 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:33705
I20260620 14:52:15.785553 28062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:33705 every 8 connection(s)
I20260620 14:52:15.786582 27935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:52:15.796840 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 27935
I20260620 14:52:15.797000 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:52:15.797418 9079 external_mini_cluster.cc:1428] Setting key 76bca9727ba0f28c0978921c647e5b07
I20260620 14:52:15.797672 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:15.788959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:55725 (local address 127.8.221.254:33607)
0620 14:52:15.789150 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:15.789155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:15.790415 (+ 1260us) server_negotiation.cc:407] Connection header received
0620 14:52:15.792015 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:15.792020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:15.792094 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:15.792201 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:15.793275 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:15.793875 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:15.795272 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:15.795531 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:15.795616 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:15.797090 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:15.797094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:15.797097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:15.797110 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:15.797130 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:15.797140 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:15.797209 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:15.797443 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 14:52:15.797519 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 14:52:15.798967 28065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:15.789392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:33607 (local address 127.8.221.194:55725)
0620 14:52:15.790215 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:15.790268 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:15.791768 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:15.792400 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:15.792407 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:15.792503 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:15.793137 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:15.793151 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:15.794001 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:15.794004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:15.795084 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:15.795094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:15.795677 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:15.795679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:15.795843 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:15.796709 (+ 866us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:52:15.796711 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:15.796757 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:52:15.796901 (+ 144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:15.797250 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:15.797261 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:15.797269 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:52:15.797435 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 14:52:15.797787 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":108,"threads_started":1}
I20260620 14:52:15.799747 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33607
--builtin_ntp_servers=127.8.221.212:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:52:15.800416 28063 heartbeater.cc:347] Connected to a master server at 127.8.221.254:33607
I20260620 14:52:15.800725 28063 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:15.801283 28063 heartbeater.cc:511] Master 127.8.221.254:33607 requested a full tablet report, sending...
I20260620 14:52:15.802348 27744 ts_manager.cc:194] Registered new tserver with Master: eab2a14e7843454fb05959d3d7a7fa46 (127.8.221.194:33705)
I20260620 14:52:15.802966 27744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.221.194:55725
W20260620 14:52:15.929915 28066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:15.930217 28066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:15.930259 28066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:15.934536 28066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:15.934613 28066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:15.934703 28066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:52:15.939917 28066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:15.941226 28066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:15.942986 28066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:15.955107 28074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:15.955178 28072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:15.955135 28071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:15.955358 28066 server_base.cc:1034] running on GCE node
I20260620 14:52:15.956027 28066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:15.956843 28066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:15.958076 28066 hybrid_clock.cc:648] HybridClock initialized: now 1781967135958054 us; error 32 us; skew 500 ppm
I20260620 14:52:15.960399 28066 webserver.cc:460] Webserver started at http://127.8.221.195:35479/ using document root <none> and password file <none>
I20260620 14:52:15.961149 28066 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:15.961220 28066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:15.961485 28066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:15.964025 28066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "d5c8477cc1734d6488e6a15be98b0dc4"
format_stamp: "Formatted at 2026-06-20 14:52:15 on dist-test-slave-65tz"
server_key: "fae6cdd611d6fcc216335198dc67025f"
server_key_iv: "47bd79fa79f9d8ec4eddd4dc534def3b"
server_key_version: "encryptionkey@0"
I20260620 14:52:15.964684 28066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "d5c8477cc1734d6488e6a15be98b0dc4"
format_stamp: "Formatted at 2026-06-20 14:52:15 on dist-test-slave-65tz"
server_key: "fae6cdd611d6fcc216335198dc67025f"
server_key_iv: "47bd79fa79f9d8ec4eddd4dc534def3b"
server_key_version: "encryptionkey@0"
I20260620 14:52:15.969305 28066 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 14:52:15.972508 28080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:15.974184 28066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:15.974335 28066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "d5c8477cc1734d6488e6a15be98b0dc4"
format_stamp: "Formatted at 2026-06-20 14:52:15 on dist-test-slave-65tz"
server_key: "fae6cdd611d6fcc216335198dc67025f"
server_key_iv: "47bd79fa79f9d8ec4eddd4dc534def3b"
server_key_version: "encryptionkey@0"
I20260620 14:52:15.974472 28066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:16.005525 28066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:16.006520 28066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:16.006762 28066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:16.007516 28066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:16.008738 28066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:16.008816 28066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:16.008895 28066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:16.008939 28066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:16.020402 28066 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:44425
I20260620 14:52:16.020459 28193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:44425 every 8 connection(s)
I20260620 14:52:16.021555 28066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:52:16.028561 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 28066
I20260620 14:52:16.028779 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:52:16.029237 9079 external_mini_cluster.cc:1428] Setting key d0cce7fc3bfcd6e83c197bb2f64d2875
I20260620 14:52:16.033617 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:16.023844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:48363 (local address 127.8.221.254:33607)
0620 14:52:16.024014 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:16.024019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:16.025363 (+ 1344us) server_negotiation.cc:407] Connection header received
0620 14:52:16.026935 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:16.026939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:16.027012 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:16.027135 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:16.028164 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.028701 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.030989 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.031340 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.031460 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:16.033044 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:16.033048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:16.033050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:16.033063 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:16.033086 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:16.033094 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:16.033184 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:16.033436 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 14:52:16.033497 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:52:16.034902 28196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:16.024373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:33607 (local address 127.8.221.195:48363)
0620 14:52:16.025158 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:16.025203 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:16.026657 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:16.027312 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:16.027319 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:16.027418 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:16.027931 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:16.027943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.028913 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.028919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:16.030792 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:16.030811 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.031548 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.031552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:16.031737 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:16.032647 (+ 910us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:52:16.032649 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:16.032705 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:52:16.032796 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:16.033227 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:16.033236 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:16.033241 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:52:16.033392 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 14:52:16.033743 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":139,"threads_started":1}
I20260620 14:52:16.036350 28194 heartbeater.cc:347] Connected to a master server at 127.8.221.254:33607
I20260620 14:52:16.036610 28194 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:16.037186 28194 heartbeater.cc:511] Master 127.8.221.254:33607 requested a full tablet report, sending...
I20260620 14:52:16.038400 27744 ts_manager.cc:194] Registered new tserver with Master: d5c8477cc1734d6488e6a15be98b0dc4 (127.8.221.195:44425)
I20260620 14:52:16.038997 27744 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.221.195:48363
I20260620 14:52:16.044134 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:52:16.051082 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:16.046572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44768 (local address 127.8.221.254:33607)
0620 14:52:16.046753 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:16.046760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:16.046973 (+ 213us) server_negotiation.cc:407] Connection header received
0620 14:52:16.047144 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:16.047147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:16.047209 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:16.047297 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:16.047663 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.048240 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.049221 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.049462 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.049577 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:16.050502 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:16.050506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:16.050509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:16.050522 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:16.050551 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:16.050559 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:16.050698 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:16.050850 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 14:52:16.050899 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:52:16.054818 27744 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:52:16.058477 27744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:52:16.075927 28210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:16.070447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:33705 (local address 127.0.0.1:44166)
0620 14:52:16.070776 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:16.070792 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:16.070900 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:16.071516 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:16.071520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:16.071540 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:16.071627 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:16.071633 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.072987 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.072994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:16.074168 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:16.074179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.075258 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.075265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:16.075394 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:16.075437 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:52:16.075559 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 14:52:16.075711 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":102,"threads_started":1}
I20260620 14:52:16.076325 28208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:16.069254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60728 (local address 127.8.221.195:44425)
0620 14:52:16.069781 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:16.069825 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:16.069861 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:52:16.070125 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:16.070136 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:16.070378 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:16.070541 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:16.071126 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.072389 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.074573 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.075341 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.075460 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:16.075640 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:16.076014 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 14:52:16.076131 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":182,"threads_started":1}
I20260620 14:52:16.076539 28206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:16.069036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57448 (local address 127.8.221.193:46241)
0620 14:52:16.070327 (+ 1291us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:16.070334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:16.070361 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:52:16.070437 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:16.070445 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:16.070613 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:16.070761 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:16.071177 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.072237 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.074592 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.075226 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.075352 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:16.075575 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:16.076233 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 14:52:16.076371 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1174,"thread_start_us":88,"threads_started":1}
I20260620 14:52:16.076812 28209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:16.070117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44166 (local address 127.8.221.194:33705)
0620 14:52:16.070927 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:16.070941 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:16.070966 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:52:16.071079 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:16.071085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:16.071326 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:16.071513 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:16.071763 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.072884 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.074300 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.075097 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.075235 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:16.075478 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:16.076414 (+ 936us) server_negotiation.cc:299] Negotiation successful
0620 14:52:16.076581 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":113,"threads_started":1}
I20260620 14:52:16.076138 28205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:16.068924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:46241 (local address 127.0.0.1:57448)
0620 14:52:16.069968 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:16.069986 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:16.070071 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:16.070908 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:16.070911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:16.070946 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:16.071038 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:16.071046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.072395 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.072398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:16.074430 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:16.074442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.075638 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.075642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:16.075737 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:16.075822 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 14:52:16.075896 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:52:16.075978 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":519,"thread_start_us":118,"threads_started":1}
I20260620 14:52:16.077181 28207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:16.068924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:44425 (local address 127.0.0.1:60728)
0620 14:52:16.069575 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:16.069625 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:16.069833 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:16.070746 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:16.070752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:16.070790 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:16.070976 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:16.070985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.072513 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.072519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:16.074430 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:16.074442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.075564 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.075568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:16.075855 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:16.075877 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:52:16.075976 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:52:16.076076 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":128,"thread_start_us":124,"threads_started":1}
I20260620 14:52:16.080627 28128 tablet_service.cc:1467] Processing CreateTablet for tablet 7676d2602e3745e1bea19ac565196c6b (DEFAULT_TABLE table=test-table [id=af31d7f3092f49dd8236adbbf4e34b51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:16.080878 27866 tablet_service.cc:1467] Processing CreateTablet for tablet 7676d2602e3745e1bea19ac565196c6b (DEFAULT_TABLE table=test-table [id=af31d7f3092f49dd8236adbbf4e34b51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:16.080878 27997 tablet_service.cc:1467] Processing CreateTablet for tablet 7676d2602e3745e1bea19ac565196c6b (DEFAULT_TABLE table=test-table [id=af31d7f3092f49dd8236adbbf4e34b51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:16.081779 28128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7676d2602e3745e1bea19ac565196c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:16.082108 27997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7676d2602e3745e1bea19ac565196c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:16.082108 27866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7676d2602e3745e1bea19ac565196c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:16.088748 28211 tablet_bootstrap.cc:492] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4: Bootstrap starting.
I20260620 14:52:16.090328 28212 tablet_bootstrap.cc:492] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46: Bootstrap starting.
I20260620 14:52:16.091506 28213 tablet_bootstrap.cc:492] T 7676d2602e3745e1bea19ac565196c6b P 10c9422e5dc14569a2860051a850326c: Bootstrap starting.
I20260620 14:52:16.091699 28211 tablet_bootstrap.cc:654] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:16.092641 28212 tablet_bootstrap.cc:654] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:16.092682 28211 log.cc:824] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:16.093475 28212 log.cc:824] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:16.093896 28213 tablet_bootstrap.cc:654] T 7676d2602e3745e1bea19ac565196c6b P 10c9422e5dc14569a2860051a850326c: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:16.094913 28213 log.cc:824] T 7676d2602e3745e1bea19ac565196c6b P 10c9422e5dc14569a2860051a850326c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:16.095245 28211 tablet_bootstrap.cc:492] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4: No bootstrap required, opened a new log
I20260620 14:52:16.095466 28211 ts_tablet_manager.cc:1397] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260620 14:52:16.095671 28212 tablet_bootstrap.cc:492] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46: No bootstrap required, opened a new log
I20260620 14:52:16.095947 28212 ts_tablet_manager.cc:1397] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260620 14:52:16.097275 28213 tablet_bootstrap.cc:492] T 7676d2602e3745e1bea19ac565196c6b P 10c9422e5dc14569a2860051a850326c: No bootstrap required, opened a new log
I20260620 14:52:16.097540 28213 ts_tablet_manager.cc:1397] T 7676d2602e3745e1bea19ac565196c6b P 10c9422e5dc14569a2860051a850326c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260620 14:52:16.100559 28212 raft_consensus.cc:348] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9422e5dc14569a2860051a850326c" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 46241 } } peers { permanent_uuid: "d5c8477cc1734d6488e6a15be98b0dc4" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44425 } } peers { permanent_uuid: "eab2a14e7843454fb05959d3d7a7fa46" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 33705 } }
I20260620 14:52:16.100522 28211 raft_consensus.cc:348] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9422e5dc14569a2860051a850326c" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 46241 } } peers { permanent_uuid: "d5c8477cc1734d6488e6a15be98b0dc4" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44425 } } peers { permanent_uuid: "eab2a14e7843454fb05959d3d7a7fa46" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 33705 } }
I20260620 14:52:16.100844 28211 raft_consensus.cc:374] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:16.100900 28212 raft_consensus.cc:374] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:16.100937 28211 raft_consensus.cc:729] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c8477cc1734d6488e6a15be98b0dc4, State: Initialized, Role: FOLLOWER
I20260620 14:52:16.100987 28212 raft_consensus.cc:729] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eab2a14e7843454fb05959d3d7a7fa46, State: Initialized, Role: FOLLOWER
I20260620 14:52:16.101292 28213 raft_consensus.cc:348] T 7676d2602e3745e1bea19ac565196c6b P 10c9422e5dc14569a2860051a850326c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9422e5dc14569a2860051a850326c" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 46241 } } peers { permanent_uuid: "d5c8477cc1734d6488e6a15be98b0dc4" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44425 } } peers { permanent_uuid: "eab2a14e7843454fb05959d3d7a7fa46" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 33705 } }
I20260620 14:52:16.101568 28213 raft_consensus.cc:374] T 7676d2602e3745e1bea19ac565196c6b P 10c9422e5dc14569a2860051a850326c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:16.101491 28211 consensus_queue.cc:260] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9422e5dc14569a2860051a850326c" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 46241 } } peers { permanent_uuid: "d5c8477cc1734d6488e6a15be98b0dc4" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44425 } } peers { permanent_uuid: "eab2a14e7843454fb05959d3d7a7fa46" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 33705 } }
I20260620 14:52:16.101491 28212 consensus_queue.cc:260] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9422e5dc14569a2860051a850326c" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 46241 } } peers { permanent_uuid: "d5c8477cc1734d6488e6a15be98b0dc4" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44425 } } peers { permanent_uuid: "eab2a14e7843454fb05959d3d7a7fa46" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 33705 } }
I20260620 14:52:16.101666 28213 raft_consensus.cc:729] T 7676d2602e3745e1bea19ac565196c6b P 10c9422e5dc14569a2860051a850326c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c9422e5dc14569a2860051a850326c, State: Initialized, Role: FOLLOWER
I20260620 14:52:16.102213 28213 consensus_queue.cc:260] T 7676d2602e3745e1bea19ac565196c6b P 10c9422e5dc14569a2860051a850326c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9422e5dc14569a2860051a850326c" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 46241 } } peers { permanent_uuid: "d5c8477cc1734d6488e6a15be98b0dc4" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44425 } } peers { permanent_uuid: "eab2a14e7843454fb05959d3d7a7fa46" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 33705 } }
I20260620 14:52:16.102329 28194 heartbeater.cc:503] Master 127.8.221.254:33607 was elected leader, sending a full tablet report...
I20260620 14:52:16.102635 28211 ts_tablet_manager.cc:1428] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260620 14:52:16.102980 28063 heartbeater.cc:503] Master 127.8.221.254:33607 was elected leader, sending a full tablet report...
I20260620 14:52:16.103402 28213 ts_tablet_manager.cc:1428] T 7676d2602e3745e1bea19ac565196c6b P 10c9422e5dc14569a2860051a850326c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:52:16.103992 27932 heartbeater.cc:503] Master 127.8.221.254:33607 was elected leader, sending a full tablet report...
I20260620 14:52:16.105286 28212 ts_tablet_manager.cc:1428] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20260620 14:52:16.126945 28217 raft_consensus.cc:482] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:52:16.127213 28217 raft_consensus.cc:504] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9422e5dc14569a2860051a850326c" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 46241 } } peers { permanent_uuid: "d5c8477cc1734d6488e6a15be98b0dc4" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44425 } } peers { permanent_uuid: "eab2a14e7843454fb05959d3d7a7fa46" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 33705 } }
I20260620 14:52:16.128479 28217 leader_election.cc:288] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10c9422e5dc14569a2860051a850326c (127.8.221.193:46241), eab2a14e7843454fb05959d3d7a7fa46 (127.8.221.194:33705)
I20260620 14:52:16.132382 28206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:16.128942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:60871 (local address 127.8.221.193:46241)
0620 14:52:16.129131 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:16.129135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:16.129152 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:52:16.129210 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:16.129213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:16.129271 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:16.129365 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:16.129649 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.130247 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.131376 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.131798 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.131932 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:16.131986 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:16.132177 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 14:52:16.132246 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:52:16.132383 28196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:16.128682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:46241 (local address 127.8.221.195:60871)
0620 14:52:16.128817 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:16.128835 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:16.128933 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:16.129379 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:16.129382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:16.129400 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:16.129526 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:16.129533 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.130375 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.130378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:16.131242 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:16.131252 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.131928 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.131932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:16.132047 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:16.132068 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:52:16.132160 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 14:52:16.132246 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 14:52:16.132871 28209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:16.129024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:35229 (local address 127.8.221.194:33705)
0620 14:52:16.129209 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:16.129214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:16.129470 (+ 256us) server_negotiation.cc:407] Connection header received
0620 14:52:16.129647 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:16.129650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:16.129711 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:16.129798 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:16.130111 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.130686 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.131753 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.132180 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.132309 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:16.132366 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:16.132619 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 14:52:16.132719 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 14:52:16.133150 27886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7676d2602e3745e1bea19ac565196c6b" candidate_uuid: "d5c8477cc1734d6488e6a15be98b0dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c9422e5dc14569a2860051a850326c" is_pre_election: true
I20260620 14:52:16.133317 28220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:16.128912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:33705 (local address 127.8.221.195:35229)
0620 14:52:16.129362 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:16.129376 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:16.129534 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:16.129834 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:16.129837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:16.129853 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:16.130000 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:16.130006 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.130797 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.130803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:16.131613 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:16.131621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.132299 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.132302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:16.132416 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:16.132475 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 14:52:16.133066 (+ 591us) client_negotiation.cc:240] Negotiation successful
0620 14:52:16.133164 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":231,"threads_started":1}
I20260620 14:52:16.133502 27886 raft_consensus.cc:2454] T 7676d2602e3745e1bea19ac565196c6b P 10c9422e5dc14569a2860051a850326c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5c8477cc1734d6488e6a15be98b0dc4 in term 0.
I20260620 14:52:16.134097 28083 leader_election.cc:302] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10c9422e5dc14569a2860051a850326c, d5c8477cc1734d6488e6a15be98b0dc4; no voters:
I20260620 14:52:16.134069 28017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7676d2602e3745e1bea19ac565196c6b" candidate_uuid: "d5c8477cc1734d6488e6a15be98b0dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eab2a14e7843454fb05959d3d7a7fa46" is_pre_election: true
I20260620 14:52:16.134379 28217 raft_consensus.cc:2781] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:52:16.134472 28217 raft_consensus.cc:482] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:52:16.134454 28017 raft_consensus.cc:2454] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5c8477cc1734d6488e6a15be98b0dc4 in term 0.
I20260620 14:52:16.134567 28217 raft_consensus.cc:3037] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:16.135802 28217 raft_consensus.cc:504] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9422e5dc14569a2860051a850326c" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 46241 } } peers { permanent_uuid: "d5c8477cc1734d6488e6a15be98b0dc4" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44425 } } peers { permanent_uuid: "eab2a14e7843454fb05959d3d7a7fa46" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 33705 } }
I20260620 14:52:16.136250 28217 leader_election.cc:288] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [CANDIDATE]: Term 1 election: Requested vote from peers 10c9422e5dc14569a2860051a850326c (127.8.221.193:46241), eab2a14e7843454fb05959d3d7a7fa46 (127.8.221.194:33705)
I20260620 14:52:16.136633 27886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7676d2602e3745e1bea19ac565196c6b" candidate_uuid: "d5c8477cc1734d6488e6a15be98b0dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10c9422e5dc14569a2860051a850326c"
I20260620 14:52:16.136762 28017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7676d2602e3745e1bea19ac565196c6b" candidate_uuid: "d5c8477cc1734d6488e6a15be98b0dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eab2a14e7843454fb05959d3d7a7fa46"
I20260620 14:52:16.136816 27886 raft_consensus.cc:3037] T 7676d2602e3745e1bea19ac565196c6b P 10c9422e5dc14569a2860051a850326c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:16.136982 28017 raft_consensus.cc:3037] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:16.138234 28017 raft_consensus.cc:2454] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c8477cc1734d6488e6a15be98b0dc4 in term 1.
I20260620 14:52:16.138335 27886 raft_consensus.cc:2454] T 7676d2602e3745e1bea19ac565196c6b P 10c9422e5dc14569a2860051a850326c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5c8477cc1734d6488e6a15be98b0dc4 in term 1.
I20260620 14:52:16.138686 28084 leader_election.cc:302] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5c8477cc1734d6488e6a15be98b0dc4, eab2a14e7843454fb05959d3d7a7fa46; no voters:
I20260620 14:52:16.138947 28217 raft_consensus.cc:2781] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:16.139267 28217 raft_consensus.cc:686] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [term 1 LEADER]: Becoming Leader. State: Replica: d5c8477cc1734d6488e6a15be98b0dc4, State: Running, Role: LEADER
I20260620 14:52:16.139712 28217 consensus_queue.cc:237] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9422e5dc14569a2860051a850326c" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 46241 } } peers { permanent_uuid: "d5c8477cc1734d6488e6a15be98b0dc4" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44425 } } peers { permanent_uuid: "eab2a14e7843454fb05959d3d7a7fa46" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 33705 } }
I20260620 14:52:16.143659 27743 catalog_manager.cc:5495] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5c8477cc1734d6488e6a15be98b0dc4 (127.8.221.195). New cstate: current_term: 1 leader_uuid: "d5c8477cc1734d6488e6a15be98b0dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10c9422e5dc14569a2860051a850326c" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 46241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5c8477cc1734d6488e6a15be98b0dc4" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 44425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eab2a14e7843454fb05959d3d7a7fa46" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260620 14:52:16.165040 28208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:16.161131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.8.221.195:44425)
0620 14:52:16.161329 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:16.161337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:16.161360 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:52:16.161448 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:16.161451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:16.161519 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:16.161647 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:16.162056 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.162643 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.163626 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.163831 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.163915 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:16.164187 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:16.164617 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:16.164730 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:16.164867 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 14:52:16.164901 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:52:16.171725 28017 raft_consensus.cc:1261] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46 [term 1 FOLLOWER]: Refusing update from remote peer d5c8477cc1734d6488e6a15be98b0dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:16.171916 27886 raft_consensus.cc:1261] T 7676d2602e3745e1bea19ac565196c6b P 10c9422e5dc14569a2860051a850326c [term 1 FOLLOWER]: Refusing update from remote peer d5c8477cc1734d6488e6a15be98b0dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:16.172596 28221 consensus_queue.cc:1035] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eab2a14e7843454fb05959d3d7a7fa46" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:16.172859 28217 consensus_queue.cc:1035] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10c9422e5dc14569a2860051a850326c" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 46241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:16.191660 27744 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:16.191919 27744 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:16.195022 27744 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0895ff7ad63248d99a694d6f1ec4d97c: Sending DeleteTablet for 3 replicas of tablet 7676d2602e3745e1bea19ac565196c6b
I20260620 14:52:16.196048 28128 tablet_service.cc:1514] Processing DeleteTablet for tablet 7676d2602e3745e1bea19ac565196c6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:16 UTC) from {username='slave'} at 127.0.0.1:60728
I20260620 14:52:16.196187 27866 tablet_service.cc:1514] Processing DeleteTablet for tablet 7676d2602e3745e1bea19ac565196c6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:16 UTC) from {username='slave'} at 127.0.0.1:57448
I20260620 14:52:16.196259 27997 tablet_service.cc:1514] Processing DeleteTablet for tablet 7676d2602e3745e1bea19ac565196c6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:16 UTC) from {username='slave'} at 127.0.0.1:44166
I20260620 14:52:16.197147 28234 tablet_replica.cc:330] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4: stopping tablet replica
I20260620 14:52:16.197397 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 27804
I20260620 14:52:16.197487 28234 raft_consensus.cc:2229] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:52:16.198395 28234 raft_consensus.cc:2258] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:16.198820 28235 tablet_replica.cc:330] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46: stopping tablet replica
I20260620 14:52:16.199105 28235 raft_consensus.cc:2229] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:16.199402 28235 raft_consensus.cc:2258] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:16.201764 28234 ts_tablet_manager.cc:1905] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:16.202903 28235 ts_tablet_manager.cc:1905] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:16.204695 28234 ts_tablet_manager.cc:1918] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:16.204906 28234 log.cc:1196] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/7676d2602e3745e1bea19ac565196c6b
I20260620 14:52:16.205333 28234 ts_tablet_manager.cc:1939] T 7676d2602e3745e1bea19ac565196c6b P d5c8477cc1734d6488e6a15be98b0dc4: Deleting consensus metadata
I20260620 14:52:16.206674 27731 catalog_manager.cc:4841] TS d5c8477cc1734d6488e6a15be98b0dc4 (127.8.221.195:44425): tablet 7676d2602e3745e1bea19ac565196c6b (table test-table [id=af31d7f3092f49dd8236adbbf4e34b51]) successfully deleted
I20260620 14:52:16.206907 28235 ts_tablet_manager.cc:1918] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:16.207084 28235 log.cc:1196] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/7676d2602e3745e1bea19ac565196c6b
I20260620 14:52:16.207441 28235 ts_tablet_manager.cc:1939] T 7676d2602e3745e1bea19ac565196c6b P eab2a14e7843454fb05959d3d7a7fa46: Deleting consensus metadata
W20260620 14:52:16.208673 27730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.221.193:46241 (error 108)
I20260620 14:52:16.208954 27731 catalog_manager.cc:4841] TS eab2a14e7843454fb05959d3d7a7fa46 (127.8.221.194:33705): tablet 7676d2602e3745e1bea19ac565196c6b (table test-table [id=af31d7f3092f49dd8236adbbf4e34b51]) successfully deleted
I20260620 14:52:16.210417 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 27935
I20260620 14:52:16.214917 28207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:16.214433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:46241 (local address 127.0.0.1:57450)
0620 14:52:16.214699 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:16.214790 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.193:46241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260620 14:52:16.215137 27730 catalog_manager.cc:4584] TS 10c9422e5dc14569a2860051a850326c (127.8.221.193:46241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7676d2602e3745e1bea19ac565196c6b: Network error: Client connection negotiation failed: client connection to 127.8.221.193:46241: connect: Connection refused (error 111)
I20260620 14:52:16.218379 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 28066
I20260620 14:52:16.225270 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 27713
2026-06-20T14:52:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1175 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:16 dist-test-slave-65tz krb5kdc[28240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:16 dist-test-slave-65tz krb5kdc[28240](info): set up 2 sockets
Jun 20 14:52:16 dist-test-slave-65tz krb5kdc[28240](info): commencing operation
krb5kdc: starting...
W20260620 14:52:17.318244 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:17 dist-test-slave-65tz krb5kdc[28240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:52:17.449357 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:43585
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:40351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:43585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:17.578902 28253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:17.579272 28253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:17.579356 28253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:17.583619 28253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:17.583760 28253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:17.583796 28253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:17.583833 28253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:17.583873 28253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:17.589885 28253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:40351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:43585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:43585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:17.591395 28253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:17.592720 28253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:17.602686 28259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:17.602664 28258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:17.602664 28261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:17.602941 28253 server_base.cc:1034] running on GCE node
I20260620 14:52:17.603686 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:17.605041 28253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:17.606246 28253 hybrid_clock.cc:648] HybridClock initialized: now 1781967137606213 us; error 52 us; skew 500 ppm
Jun 20 14:52:17 dist-test-slave-65tz krb5kdc[28240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967137, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:17.609416 28253 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:52:17.610589 28253 webserver.cc:460] Webserver started at http://127.8.221.254:43813/ using document root <none> and password file <none>
I20260620 14:52:17.611300 28253 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:17.611402 28253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:17.611672 28253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:17.614280 28253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "2559f24ad9d34f788ebb5b61a282e715"
format_stamp: "Formatted at 2026-06-20 14:52:17 on dist-test-slave-65tz"
server_key: "06bd3c78f188a78ef79bbdf48b3c97ad"
server_key_iv: "8fc8f9f68e42ae2d339cbce65096e092"
server_key_version: "encryptionkey@0"
I20260620 14:52:17.614929 28253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "2559f24ad9d34f788ebb5b61a282e715"
format_stamp: "Formatted at 2026-06-20 14:52:17 on dist-test-slave-65tz"
server_key: "06bd3c78f188a78ef79bbdf48b3c97ad"
server_key_iv: "8fc8f9f68e42ae2d339cbce65096e092"
server_key_version: "encryptionkey@0"
I20260620 14:52:17.619336 28253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:52:17.622252 28268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:17.623852 28253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:52:17.624012 28253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "2559f24ad9d34f788ebb5b61a282e715"
format_stamp: "Formatted at 2026-06-20 14:52:17 on dist-test-slave-65tz"
server_key: "06bd3c78f188a78ef79bbdf48b3c97ad"
server_key_iv: "8fc8f9f68e42ae2d339cbce65096e092"
server_key_version: "encryptionkey@0"
I20260620 14:52:17.624140 28253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:17.651717 28253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:17.656015 28253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:17.656260 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:17.665264 28253 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:43585
I20260620 14:52:17.665293 28320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:43585 every 8 connection(s)
I20260620 14:52:17.666644 28253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:17.667130 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 28253
I20260620 14:52:17.667309 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:17.667673 9079 external_mini_cluster.cc:1428] Setting key 2c971652dba28da4ddb197dea116bd87
I20260620 14:52:17.670550 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 14:52:17 dist-test-slave-65tz krb5kdc[28240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:52:17.677697 28321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715: Bootstrap starting.
I20260620 14:52:17.680586 28321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:17.681455 28321 log.cc:824] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:17.682966 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:17.669354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42406 (local address 127.8.221.254:43585)
0620 14:52:17.669827 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:17.669858 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:17.669897 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:52:17.670610 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:17.670625 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:17.671038 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:17.671398 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:17.672213 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:17.673257 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:17.674203 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:17.674692 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:17.674874 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:17.678080 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:17.678085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:17.678097 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:17.678133 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:17.679887 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:17.680333 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:17.680339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:17.680343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:17.680454 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:17.680758 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:17.680762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:17.680763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:17.681207 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:17.681417 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:17.681681 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 14:52:17.682054 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":166,"threads_started":1}
I20260620 14:52:17.683952 28321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715: No bootstrap required, opened a new log
I20260620 14:52:17.688308 28321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2559f24ad9d34f788ebb5b61a282e715" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43585 } }
I20260620 14:52:17.688658 28321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:17.688746 28321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2559f24ad9d34f788ebb5b61a282e715, State: Initialized, Role: FOLLOWER
I20260620 14:52:17.689261 28321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2559f24ad9d34f788ebb5b61a282e715" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43585 } }
I20260620 14:52:17.689405 28321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:17.689498 28321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:17.689627 28321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:17.690739 28321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2559f24ad9d34f788ebb5b61a282e715" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43585 } }
I20260620 14:52:17.691129 28321 leader_election.cc:302] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2559f24ad9d34f788ebb5b61a282e715; no voters:
I20260620 14:52:17.691521 28321 leader_election.cc:288] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:17.691694 28326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:17.692077 28326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715 [term 1 LEADER]: Becoming Leader. State: Replica: 2559f24ad9d34f788ebb5b61a282e715, State: Running, Role: LEADER
I20260620 14:52:17.692519 28326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2559f24ad9d34f788ebb5b61a282e715" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43585 } }
I20260620 14:52:17.692960 28321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:17.694332 28328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2559f24ad9d34f788ebb5b61a282e715. Latest consensus state: current_term: 1 leader_uuid: "2559f24ad9d34f788ebb5b61a282e715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2559f24ad9d34f788ebb5b61a282e715" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43585 } } }
I20260620 14:52:17.694476 28328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:17.694856 28327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2559f24ad9d34f788ebb5b61a282e715" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2559f24ad9d34f788ebb5b61a282e715" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43585 } } }
I20260620 14:52:17.694968 28327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715 [sys.catalog]: This master's current role is: LEADER
W20260620 14:52:17.699124 28341 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:52:17.699215 28341 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:52:17.699303 28335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:17.700182 28335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:17.706868 28335 catalog_manager.cc:1347] Generated new cluster ID: 07da2fc9dc4c4c4688016d0615c8ef8e
I20260620 14:52:17.706964 28335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:17.719779 28335 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:17.721123 28335 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:17.727720 28335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715: Generated new TSK 0
I20260620 14:52:17.728935 28335 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.221.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
Entry for principal kudu/127.8.221.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab.
I20260620 14:52:17.765482 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:43585
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:40351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:17.898718 28347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:17.899039 28347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:17.899088 28347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:17.903421 28347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:17.903501 28347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:17.903594 28347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:52:17.909523 28347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:40351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.193.keytab
--principal=kudu/127.8.221.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:43585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:17.911033 28347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:17.912351 28347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:17.923914 28353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:17.923818 28352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:17.923830 28355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:17.924242 28347 server_base.cc:1034] running on GCE node
I20260620 14:52:17.924700 28347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:17.925460 28347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:17.926661 28347 hybrid_clock.cc:648] HybridClock initialized: now 1781967137926612 us; error 66 us; skew 500 ppm
Jun 20 14:52:17 dist-test-slave-65tz krb5kdc[28240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967137, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:17.929975 28347 init.cc:377] Logged in from keytab as kudu/127.8.221.193@KRBTEST.COM (short username kudu)
I20260620 14:52:17.930946 28347 webserver.cc:460] Webserver started at http://127.8.221.193:36905/ using document root <none> and password file <none>
I20260620 14:52:17.931576 28347 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:17.931675 28347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:17.931905 28347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:17.934100 28347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "f8e38a8b1599499c92e259b76b8efef9"
format_stamp: "Formatted at 2026-06-20 14:52:17 on dist-test-slave-65tz"
server_key: "e45c850fa3223c09d94e6e8fd059b60a"
server_key_iv: "3e1b4ee1b06ef703faccca87b5271dd4"
server_key_version: "encryptionkey@0"
I20260620 14:52:17.934759 28347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "f8e38a8b1599499c92e259b76b8efef9"
format_stamp: "Formatted at 2026-06-20 14:52:17 on dist-test-slave-65tz"
server_key: "e45c850fa3223c09d94e6e8fd059b60a"
server_key_iv: "3e1b4ee1b06ef703faccca87b5271dd4"
server_key_version: "encryptionkey@0"
I20260620 14:52:17.939491 28347 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:52:17.944013 28362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:17.946251 28347 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 14:52:17.946456 28347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "f8e38a8b1599499c92e259b76b8efef9"
format_stamp: "Formatted at 2026-06-20 14:52:17 on dist-test-slave-65tz"
server_key: "e45c850fa3223c09d94e6e8fd059b60a"
server_key_iv: "3e1b4ee1b06ef703faccca87b5271dd4"
server_key_version: "encryptionkey@0"
I20260620 14:52:17.946641 28347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:18.001365 28347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:18.005582 28347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:18.005867 28347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:18.006623 28347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:18.007640 28347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:18.007718 28347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:18.007800 28347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:18.007853 28347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:18.019390 28347 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:33039
I20260620 14:52:18.019429 28475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:33039 every 8 connection(s)
I20260620 14:52:18.020588 28347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:52:18.024899 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 28347
I20260620 14:52:18.025079 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:52:18.025496 9079 external_mini_cluster.cc:1428] Setting key ce76af2589081623f36444a5fa739c20
Jun 20 14:52:18 dist-test-slave-65tz krb5kdc[28240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967137, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.193@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.194@KRBTEST.COM; defaulting to no policy
I20260620 14:52:18.040050 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.022906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:49135 (local address 127.8.221.254:43585)
0620 14:52:18.023134 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.023140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.024279 (+ 1139us) server_negotiation.cc:407] Connection header received
0620 14:52:18.025910 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.025914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.025968 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.026063 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:18.028098 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.028735 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.030659 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.031066 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.031205 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.035005 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:18.035010 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:18.035014 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:18.035049 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:18.036994 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:18.037698 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:18.037701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:18.037702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:18.037781 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:18.038556 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:18.038558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:18.038559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:18.038756 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:18.038847 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.039639 (+ 792us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.039838 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.194@KRBTEST.COM" created.
I20260620 14:52:18.041409 28478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.023136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43585 (local address 127.8.221.193:49135)
0620 14:52:18.024063 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:18.024117 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:18.025518 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:18.026373 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:18.026385 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:18.026948 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:18.027893 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.027918 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.028904 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.028912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.030502 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.030514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.031420 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.031429 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.031605 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.032454 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:18.032474 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:18.034782 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:18.037146 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:18.037153 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:18.037171 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:18.037588 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:18.038161 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:18.038167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:18.038173 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:18.038443 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:18.038892 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:18.038898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:18.039297 (+ 399us) client_negotiation.cc:769] Sending connection context
0620 14:52:18.039562 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 14:52:18.040053 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":151,"threads_started":1}
I20260620 14:52:18.043020 28476 heartbeater.cc:347] Connected to a master server at 127.8.221.254:43585
I20260620 14:52:18.043337 28476 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:18.044014 28476 heartbeater.cc:511] Master 127.8.221.254:43585 requested a full tablet report, sending...
I20260620 14:52:18.045931 28285 ts_manager.cc:194] Registered new tserver with Master: f8e38a8b1599499c92e259b76b8efef9 (127.8.221.193:33039)
I20260620 14:52:18.047751 28285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.193@KRBTEST.COM'} at 127.8.221.193:49135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
Entry for principal kudu/127.8.221.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab.
I20260620 14:52:18.058864 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:43585
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:40351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:18.175992 28481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:18.176290 28481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:18.176333 28481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:18.180683 28481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:18.180759 28481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:18.180821 28481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:52:18.186235 28481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:40351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.194.keytab
--principal=kudu/127.8.221.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:43585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:18.187263 28481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:18.188275 28481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:18.198802 28487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:18.198923 28489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:18.198783 28486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:18.199257 28481 server_base.cc:1034] running on GCE node
I20260620 14:52:18.199894 28481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:18.200898 28481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:18.202121 28481 hybrid_clock.cc:648] HybridClock initialized: now 1781967138202097 us; error 41 us; skew 500 ppm
Jun 20 14:52:18 dist-test-slave-65tz krb5kdc[28240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967138, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:18.205693 28481 init.cc:377] Logged in from keytab as kudu/127.8.221.194@KRBTEST.COM (short username kudu)
I20260620 14:52:18.206969 28481 webserver.cc:460] Webserver started at http://127.8.221.194:42661/ using document root <none> and password file <none>
I20260620 14:52:18.207742 28481 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:18.207854 28481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:18.208246 28481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:18.211015 28481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "c7322f96571b4b7cb4ce36781fd93525"
format_stamp: "Formatted at 2026-06-20 14:52:18 on dist-test-slave-65tz"
server_key: "28a8e56e44197d769ed1ec99ac583dc6"
server_key_iv: "502b3d3ae26bd9be0ee7d9e36b2d136e"
server_key_version: "encryptionkey@0"
I20260620 14:52:18.211740 28481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "c7322f96571b4b7cb4ce36781fd93525"
format_stamp: "Formatted at 2026-06-20 14:52:18 on dist-test-slave-65tz"
server_key: "28a8e56e44197d769ed1ec99ac583dc6"
server_key_iv: "502b3d3ae26bd9be0ee7d9e36b2d136e"
server_key_version: "encryptionkey@0"
I20260620 14:52:18.216595 28481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:52:18.219830 28496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:18.221493 28481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:52:18.221668 28481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "c7322f96571b4b7cb4ce36781fd93525"
format_stamp: "Formatted at 2026-06-20 14:52:18 on dist-test-slave-65tz"
server_key: "28a8e56e44197d769ed1ec99ac583dc6"
server_key_iv: "502b3d3ae26bd9be0ee7d9e36b2d136e"
server_key_version: "encryptionkey@0"
I20260620 14:52:18.221854 28481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:18.245330 28481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:18.249774 28481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:18.250109 28481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:18.250988 28481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:18.252156 28481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:18.252235 28481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:18.252351 28481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:18.252409 28481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:18.265075 28481 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:44195
I20260620 14:52:18.265100 28609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:44195 every 8 connection(s)
I20260620 14:52:18.266412 28481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:52:18.267395 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 28481
I20260620 14:52:18.267575 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:52:18.268000 9079 external_mini_cluster.cc:1428] Setting key 0282cf446e33575cb4fbc6b3867217ec
Jun 20 14:52:18 dist-test-slave-65tz krb5kdc[28240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967138, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.194@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.221.195@KRBTEST.COM; defaulting to no policy
I20260620 14:52:18.282570 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.268959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:49931 (local address 127.8.221.254:43585)
0620 14:52:18.269166 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.269172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.270814 (+ 1642us) server_negotiation.cc:407] Connection header received
0620 14:52:18.271830 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.271834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.271886 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.271964 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:18.273211 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.273735 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.274966 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.275198 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.275310 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.278591 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:18.278594 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:18.278596 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:18.278617 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:18.280381 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:18.280919 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:18.280923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:18.280925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:18.281004 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:18.281379 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:18.281384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:18.281385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:18.281566 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:18.281681 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.282271 (+ 590us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.282397 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.195@KRBTEST.COM" created.
I20260620 14:52:18.283604 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.269618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43585 (local address 127.8.221.194:49931)
0620 14:52:18.270644 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:18.270694 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:18.271597 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:18.272149 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:18.272156 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:18.272552 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:18.273057 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.273070 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.273924 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.273927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.274838 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.274847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.275354 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.275357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.275532 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.276325 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:18.276341 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:18.278420 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:18.280499 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:18.280507 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:18.280534 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:18.280801 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:18.281154 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:18.281157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:18.281158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:18.281281 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:18.281680 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:18.281686 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:18.281977 (+ 291us) client_negotiation.cc:769] Sending connection context
0620 14:52:18.282231 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 14:52:18.282621 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":201,"threads_started":1}
I20260620 14:52:18.285176 28610 heartbeater.cc:347] Connected to a master server at 127.8.221.254:43585
I20260620 14:52:18.285450 28610 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:18.286049 28610 heartbeater.cc:511] Master 127.8.221.254:43585 requested a full tablet report, sending...
I20260620 14:52:18.287184 28285 ts_manager.cc:194] Registered new tserver with Master: c7322f96571b4b7cb4ce36781fd93525 (127.8.221.194:44195)
I20260620 14:52:18.287783 28285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.194@KRBTEST.COM'} at 127.8.221.194:49931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
Entry for principal kudu/127.8.221.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab.
I20260620 14:52:18.301286 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:43585
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:40351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:18.430521 28615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:18.430886 28615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:18.430938 28615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:18.435474 28615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:18.435549 28615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:18.435616 28615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:52:18.441460 28615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:40351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.195.keytab
--principal=kudu/127.8.221.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:43585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:18.442866 28615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:18.444187 28615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:18.456231 28623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:18.456243 28620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:18.456609 28615 server_base.cc:1034] running on GCE node
W20260620 14:52:18.456235 28621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:18.457193 28615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:18.458218 28615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:18.459394 28615 hybrid_clock.cc:648] HybridClock initialized: now 1781967138459361 us; error 42 us; skew 500 ppm
Jun 20 14:52:18 dist-test-slave-65tz krb5kdc[28240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967138, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:18.462910 28615 init.cc:377] Logged in from keytab as kudu/127.8.221.195@KRBTEST.COM (short username kudu)
I20260620 14:52:18.464120 28615 webserver.cc:460] Webserver started at http://127.8.221.195:39229/ using document root <none> and password file <none>
I20260620 14:52:18.465082 28615 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:18.465188 28615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:18.465489 28615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:18.469182 28615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "282f6ed7531c4d54838790da367ee2db"
format_stamp: "Formatted at 2026-06-20 14:52:18 on dist-test-slave-65tz"
server_key: "c88e837eb57bfc71ad771dd6e1f47128"
server_key_iv: "4cee5c6160562a21c52b4dbb0247056c"
server_key_version: "encryptionkey@0"
I20260620 14:52:18.470021 28615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "282f6ed7531c4d54838790da367ee2db"
format_stamp: "Formatted at 2026-06-20 14:52:18 on dist-test-slave-65tz"
server_key: "c88e837eb57bfc71ad771dd6e1f47128"
server_key_iv: "4cee5c6160562a21c52b4dbb0247056c"
server_key_version: "encryptionkey@0"
I20260620 14:52:18.474941 28615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:52:18.477919 28630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:18.479597 28615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:18.479710 28615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "282f6ed7531c4d54838790da367ee2db"
format_stamp: "Formatted at 2026-06-20 14:52:18 on dist-test-slave-65tz"
server_key: "c88e837eb57bfc71ad771dd6e1f47128"
server_key_iv: "4cee5c6160562a21c52b4dbb0247056c"
server_key_version: "encryptionkey@0"
I20260620 14:52:18.479816 28615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:18.497010 28615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:18.500661 28615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:18.500845 28615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:18.501483 28615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:18.502496 28615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:18.502553 28615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:18.502599 28615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:18.502615 28615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:18.514536 28615 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:33389
I20260620 14:52:18.514559 28743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:33389 every 8 connection(s)
I20260620 14:52:18.515707 28615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:52:18.519711 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 28615
I20260620 14:52:18.519888 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:52:18.520282 9079 external_mini_cluster.cc:1428] Setting key e2a4a9549f51d65b875d37fccbde5b02
Jun 20 14:52:18 dist-test-slave-65tz krb5kdc[28240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967138, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.195@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:52:18.531629 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.518094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:38365 (local address 127.8.221.254:43585)
0620 14:52:18.518289 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.518295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.519578 (+ 1283us) server_negotiation.cc:407] Connection header received
0620 14:52:18.520798 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.520802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.520862 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.520957 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:18.522345 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.522907 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.524074 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.524302 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.524413 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.527456 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:18.527460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:18.527462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:18.527487 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:18.529179 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:18.529882 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:18.529885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:18.529886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:18.529962 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:18.530384 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:18.530387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:18.530388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:18.530575 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:18.530666 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.531387 (+ 721us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.531493 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:52:18.532756 28746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.518640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:43585 (local address 127.8.221.195:38365)
0620 14:52:18.519260 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:18.519313 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:18.520570 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:18.521164 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:18.521171 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:18.521557 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:18.522175 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.522191 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.523046 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.523050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.523941 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.523950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.524455 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.524459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.524622 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.525302 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:18.525319 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:18.527247 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:18.529334 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:18.529340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:18.529359 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:18.529712 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:18.530072 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:18.530074 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:18.530076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:18.530240 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:18.530710 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:18.530717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:18.531109 (+ 392us) client_negotiation.cc:769] Sending connection context
0620 14:52:18.531309 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 14:52:18.531661 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":137,"threads_started":1}
I20260620 14:52:18.534273 28744 heartbeater.cc:347] Connected to a master server at 127.8.221.254:43585
I20260620 14:52:18.534610 28744 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:18.535286 28744 heartbeater.cc:511] Master 127.8.221.254:43585 requested a full tablet report, sending...
I20260620 14:52:18.536459 28284 ts_manager.cc:194] Registered new tserver with Master: 282f6ed7531c4d54838790da367ee2db (127.8.221.195:33389)
I20260620 14:52:18.537112 28284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.221.195@KRBTEST.COM'} at 127.8.221.195:38365
I20260620 14:52:18.547214 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:52:18.558471 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.549716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42412 (local address 127.8.221.254:43585)
0620 14:52:18.549879 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.549882 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.550251 (+ 369us) server_negotiation.cc:407] Connection header received
0620 14:52:18.550330 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.550333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.550383 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.550461 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:18.551250 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.551886 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.552765 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.552981 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.553081 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.554743 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:18.554746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:18.554748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:18.554771 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:18.556520 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:18.557014 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:18.557019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:18.557021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:18.557111 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:18.557406 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:18.557410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:18.557412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:18.557687 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:18.557878 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.558136 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.558271 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:52:18.562254 28284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:52:18.565564 28284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:52:18.582796 28758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.576000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:33389 (local address 127.0.0.1:35322)
0620 14:52:18.576998 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:18.577049 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:18.577196 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:18.577932 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:18.577937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:18.577962 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:18.578115 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.578121 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.579437 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.579444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.580747 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.580758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.582110 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.582115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.582293 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.582354 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 14:52:18.582462 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:52:18.582617 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":4,"thread_start_us":131,"threads_started":1}
I20260620 14:52:18.583134 28757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.576261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:44195 (local address 127.0.0.1:46664)
0620 14:52:18.577584 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:18.577597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:18.577693 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:18.578558 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:18.578568 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:18.578622 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:18.578805 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.578814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.580363 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.580366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.581333 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.581342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.582402 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.582406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.582519 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.582549 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 14:52:18.582641 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 14:52:18.582734 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1266,"mutex_wait_us":32,"thread_start_us":91,"threads_started":1}
I20260620 14:52:18.583195 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.576984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46664 (local address 127.8.221.194:44195)
0620 14:52:18.577456 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.577467 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.577701 (+ 234us) server_negotiation.cc:407] Connection header received
0620 14:52:18.577879 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.577888 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.578135 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.578323 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:18.578962 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.580194 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.581481 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.582283 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.582403 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.582555 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.582677 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.582835 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":125,"threads_started":1}
I20260620 14:52:18.583438 28759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.576375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.8.221.195:33389)
0620 14:52:18.576909 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.576925 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.577358 (+ 433us) server_negotiation.cc:407] Connection header received
0620 14:52:18.577453 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.577459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.577693 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.577865 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:18.578233 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.579301 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.580907 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.581967 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.582094 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.582302 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.583143 (+ 841us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.583283 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":171,"threads_started":1}
I20260620 14:52:18.584653 28755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.576000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:33039 (local address 127.0.0.1:56044)
0620 14:52:18.577698 (+ 1698us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:18.577716 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:18.577860 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:18.578392 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:18.578395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:18.578414 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:18.578513 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.578520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.580736 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.580743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.582219 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.582231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.584164 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.584168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.584277 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.584311 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 14:52:18.584399 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:52:18.584487 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1621,"mutex_wait_us":491,"thread_start_us":94,"threads_started":1}
I20260620 14:52:18.584784 28756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.576192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56044 (local address 127.8.221.193:33039)
0620 14:52:18.577250 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.577259 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.577843 (+ 584us) server_negotiation.cc:407] Connection header received
0620 14:52:18.578042 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.578048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.578224 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.578384 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:18.578649 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.580601 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.582379 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.583823 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.583939 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.584095 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.584458 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.584638 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":918,"thread_start_us":95,"threads_started":1}
I20260620 14:52:18.586793 28678 tablet_service.cc:1467] Processing CreateTablet for tablet 708a481c0830473f958227d585ddf12f (DEFAULT_TABLE table=test-table [id=c1651b6015444623b20f686890080a04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:18.587877 28678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708a481c0830473f958227d585ddf12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:18.587944 28410 tablet_service.cc:1467] Processing CreateTablet for tablet 708a481c0830473f958227d585ddf12f (DEFAULT_TABLE table=test-table [id=c1651b6015444623b20f686890080a04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:18.587944 28544 tablet_service.cc:1467] Processing CreateTablet for tablet 708a481c0830473f958227d585ddf12f (DEFAULT_TABLE table=test-table [id=c1651b6015444623b20f686890080a04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:18.589260 28544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708a481c0830473f958227d585ddf12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:18.589269 28410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708a481c0830473f958227d585ddf12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:18.594141 28761 tablet_bootstrap.cc:492] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db: Bootstrap starting.
I20260620 14:52:18.597237 28761 tablet_bootstrap.cc:654] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:18.598022 28762 tablet_bootstrap.cc:492] T 708a481c0830473f958227d585ddf12f P f8e38a8b1599499c92e259b76b8efef9: Bootstrap starting.
I20260620 14:52:18.598518 28763 tablet_bootstrap.cc:492] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525: Bootstrap starting.
I20260620 14:52:18.598538 28761 log.cc:824] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:18.600668 28763 tablet_bootstrap.cc:654] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:18.600795 28762 tablet_bootstrap.cc:654] T 708a481c0830473f958227d585ddf12f P f8e38a8b1599499c92e259b76b8efef9: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:18.601104 28761 tablet_bootstrap.cc:492] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db: No bootstrap required, opened a new log
I20260620 14:52:18.601403 28761 ts_tablet_manager.cc:1397] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260620 14:52:18.601663 28762 log.cc:824] T 708a481c0830473f958227d585ddf12f P f8e38a8b1599499c92e259b76b8efef9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:18.601827 28763 log.cc:824] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:18.603637 28762 tablet_bootstrap.cc:492] T 708a481c0830473f958227d585ddf12f P f8e38a8b1599499c92e259b76b8efef9: No bootstrap required, opened a new log
I20260620 14:52:18.603847 28762 ts_tablet_manager.cc:1397] T 708a481c0830473f958227d585ddf12f P f8e38a8b1599499c92e259b76b8efef9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:52:18.605263 28763 tablet_bootstrap.cc:492] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525: No bootstrap required, opened a new log
I20260620 14:52:18.605482 28763 ts_tablet_manager.cc:1397] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 14:52:18.605561 28761 raft_consensus.cc:348] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e38a8b1599499c92e259b76b8efef9" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33039 } } peers { permanent_uuid: "282f6ed7531c4d54838790da367ee2db" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33389 } } peers { permanent_uuid: "c7322f96571b4b7cb4ce36781fd93525" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 44195 } }
I20260620 14:52:18.605938 28761 raft_consensus.cc:374] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:18.606002 28761 raft_consensus.cc:729] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282f6ed7531c4d54838790da367ee2db, State: Initialized, Role: FOLLOWER
I20260620 14:52:18.606526 28761 consensus_queue.cc:260] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e38a8b1599499c92e259b76b8efef9" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33039 } } peers { permanent_uuid: "282f6ed7531c4d54838790da367ee2db" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33389 } } peers { permanent_uuid: "c7322f96571b4b7cb4ce36781fd93525" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 44195 } }
I20260620 14:52:18.607462 28744 heartbeater.cc:503] Master 127.8.221.254:43585 was elected leader, sending a full tablet report...
I20260620 14:52:18.607724 28761 ts_tablet_manager.cc:1428] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 14:52:18.608115 28762 raft_consensus.cc:348] T 708a481c0830473f958227d585ddf12f P f8e38a8b1599499c92e259b76b8efef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e38a8b1599499c92e259b76b8efef9" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33039 } } peers { permanent_uuid: "282f6ed7531c4d54838790da367ee2db" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33389 } } peers { permanent_uuid: "c7322f96571b4b7cb4ce36781fd93525" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 44195 } }
I20260620 14:52:18.608385 28762 raft_consensus.cc:374] T 708a481c0830473f958227d585ddf12f P f8e38a8b1599499c92e259b76b8efef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:18.608441 28762 raft_consensus.cc:729] T 708a481c0830473f958227d585ddf12f P f8e38a8b1599499c92e259b76b8efef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e38a8b1599499c92e259b76b8efef9, State: Initialized, Role: FOLLOWER
I20260620 14:52:18.608939 28762 consensus_queue.cc:260] T 708a481c0830473f958227d585ddf12f P f8e38a8b1599499c92e259b76b8efef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e38a8b1599499c92e259b76b8efef9" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33039 } } peers { permanent_uuid: "282f6ed7531c4d54838790da367ee2db" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33389 } } peers { permanent_uuid: "c7322f96571b4b7cb4ce36781fd93525" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 44195 } }
I20260620 14:52:18.609732 28476 heartbeater.cc:503] Master 127.8.221.254:43585 was elected leader, sending a full tablet report...
I20260620 14:52:18.610010 28762 ts_tablet_manager.cc:1428] T 708a481c0830473f958227d585ddf12f P f8e38a8b1599499c92e259b76b8efef9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:52:18.610062 28763 raft_consensus.cc:348] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e38a8b1599499c92e259b76b8efef9" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33039 } } peers { permanent_uuid: "282f6ed7531c4d54838790da367ee2db" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33389 } } peers { permanent_uuid: "c7322f96571b4b7cb4ce36781fd93525" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 44195 } }
I20260620 14:52:18.610388 28763 raft_consensus.cc:374] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:18.610459 28763 raft_consensus.cc:729] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7322f96571b4b7cb4ce36781fd93525, State: Initialized, Role: FOLLOWER
I20260620 14:52:18.610911 28763 consensus_queue.cc:260] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e38a8b1599499c92e259b76b8efef9" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33039 } } peers { permanent_uuid: "282f6ed7531c4d54838790da367ee2db" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33389 } } peers { permanent_uuid: "c7322f96571b4b7cb4ce36781fd93525" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 44195 } }
I20260620 14:52:18.611843 28610 heartbeater.cc:503] Master 127.8.221.254:43585 was elected leader, sending a full tablet report...
I20260620 14:52:18.611871 28763 ts_tablet_manager.cc:1428] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 14:52:18.678706 28769 raft_consensus.cc:482] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:52:18.678984 28769 raft_consensus.cc:504] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e38a8b1599499c92e259b76b8efef9" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33039 } } peers { permanent_uuid: "282f6ed7531c4d54838790da367ee2db" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33389 } } peers { permanent_uuid: "c7322f96571b4b7cb4ce36781fd93525" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 44195 } }
I20260620 14:52:18.680430 28769 leader_election.cc:288] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e38a8b1599499c92e259b76b8efef9 (127.8.221.193:33039), 282f6ed7531c4d54838790da367ee2db (127.8.221.195:33389)
I20260620 14:52:18.684926 28759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.680877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:43681 (local address 127.8.221.195:33389)
0620 14:52:18.681020 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.681026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.681043 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:52:18.681228 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.681231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.681283 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.681380 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:18.681704 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.682384 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.683693 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.684292 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.684433 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.684490 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.684707 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.684776 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:52:18.684926 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.680780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:33389 (local address 127.8.221.194:43681)
0620 14:52:18.680942 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:18.680963 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:18.681076 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:18.681394 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:18.681397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:18.681415 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:18.681577 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.681584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.682512 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.682519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.683542 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.683559 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.684427 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.684431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.684568 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.684589 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:52:18.684690 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:52:18.684778 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":1}
I20260620 14:52:18.685113 28770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.680785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:33039 (local address 127.8.221.194:60487)
0620 14:52:18.681376 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:18.681389 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:18.681549 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:18.681844 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:18.681848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:18.681864 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:18.682021 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.682026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.682802 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.682807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.683742 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.683765 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.684605 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.684608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.684720 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.684745 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:52:18.684845 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:52:18.684927 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":176,"thread_start_us":213,"threads_started":1}
I20260620 14:52:18.685101 28756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.680914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:60487 (local address 127.8.221.193:33039)
0620 14:52:18.681069 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.681077 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.681489 (+ 412us) server_negotiation.cc:407] Connection header received
0620 14:52:18.681669 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.681672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.681727 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.681843 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:18.682134 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.682683 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.683889 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.684471 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.684617 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.684680 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.684864 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.684939 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:52:18.685707 28698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708a481c0830473f958227d585ddf12f" candidate_uuid: "c7322f96571b4b7cb4ce36781fd93525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282f6ed7531c4d54838790da367ee2db" is_pre_election: true
I20260620 14:52:18.685966 28430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708a481c0830473f958227d585ddf12f" candidate_uuid: "c7322f96571b4b7cb4ce36781fd93525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e38a8b1599499c92e259b76b8efef9" is_pre_election: true
I20260620 14:52:18.686120 28698 raft_consensus.cc:2454] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7322f96571b4b7cb4ce36781fd93525 in term 0.
I20260620 14:52:18.686328 28430 raft_consensus.cc:2454] T 708a481c0830473f958227d585ddf12f P f8e38a8b1599499c92e259b76b8efef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7322f96571b4b7cb4ce36781fd93525 in term 0.
I20260620 14:52:18.686731 28498 leader_election.cc:302] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 282f6ed7531c4d54838790da367ee2db, c7322f96571b4b7cb4ce36781fd93525; no voters:
I20260620 14:52:18.687157 28769 raft_consensus.cc:2781] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:52:18.687275 28769 raft_consensus.cc:482] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:52:18.687328 28769 raft_consensus.cc:3037] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:18.688505 28769 raft_consensus.cc:504] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e38a8b1599499c92e259b76b8efef9" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33039 } } peers { permanent_uuid: "282f6ed7531c4d54838790da367ee2db" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33389 } } peers { permanent_uuid: "c7322f96571b4b7cb4ce36781fd93525" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 44195 } }
I20260620 14:52:18.688926 28769 leader_election.cc:288] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [CANDIDATE]: Term 1 election: Requested vote from peers f8e38a8b1599499c92e259b76b8efef9 (127.8.221.193:33039), 282f6ed7531c4d54838790da367ee2db (127.8.221.195:33389)
I20260620 14:52:18.689369 28430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708a481c0830473f958227d585ddf12f" candidate_uuid: "c7322f96571b4b7cb4ce36781fd93525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e38a8b1599499c92e259b76b8efef9"
I20260620 14:52:18.689472 28698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708a481c0830473f958227d585ddf12f" candidate_uuid: "c7322f96571b4b7cb4ce36781fd93525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "282f6ed7531c4d54838790da367ee2db"
I20260620 14:52:18.689543 28430 raft_consensus.cc:3037] T 708a481c0830473f958227d585ddf12f P f8e38a8b1599499c92e259b76b8efef9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:18.689663 28698 raft_consensus.cc:3037] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:18.690840 28430 raft_consensus.cc:2454] T 708a481c0830473f958227d585ddf12f P f8e38a8b1599499c92e259b76b8efef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7322f96571b4b7cb4ce36781fd93525 in term 1.
I20260620 14:52:18.691143 28698 raft_consensus.cc:2454] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7322f96571b4b7cb4ce36781fd93525 in term 1.
I20260620 14:52:18.691272 28497 leader_election.cc:302] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7322f96571b4b7cb4ce36781fd93525, f8e38a8b1599499c92e259b76b8efef9; no voters:
I20260620 14:52:18.691537 28769 raft_consensus.cc:2781] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:18.691789 28769 raft_consensus.cc:686] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [term 1 LEADER]: Becoming Leader. State: Replica: c7322f96571b4b7cb4ce36781fd93525, State: Running, Role: LEADER
I20260620 14:52:18.692250 28769 consensus_queue.cc:237] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e38a8b1599499c92e259b76b8efef9" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33039 } } peers { permanent_uuid: "282f6ed7531c4d54838790da367ee2db" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33389 } } peers { permanent_uuid: "c7322f96571b4b7cb4ce36781fd93525" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 44195 } }
I20260620 14:52:18.696286 28283 catalog_manager.cc:5495] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7322f96571b4b7cb4ce36781fd93525 (127.8.221.194). New cstate: current_term: 1 leader_uuid: "c7322f96571b4b7cb4ce36781fd93525" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e38a8b1599499c92e259b76b8efef9" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "282f6ed7531c4d54838790da367ee2db" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7322f96571b4b7cb4ce36781fd93525" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 44195 } health_report { overall_health: HEALTHY } } }
I20260620 14:52:18.710431 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.706516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46666 (local address 127.8.221.194:44195)
0620 14:52:18.706651 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.706655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.706671 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:52:18.706817 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.706819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.706876 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.706971 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:18.707272 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.707844 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.708797 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.709032 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.709131 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.709438 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:18.709990 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:18.710145 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.710211 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.710266 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:52:18.717957 28430 raft_consensus.cc:1261] T 708a481c0830473f958227d585ddf12f P f8e38a8b1599499c92e259b76b8efef9 [term 1 FOLLOWER]: Refusing update from remote peer c7322f96571b4b7cb4ce36781fd93525: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:18.718097 28698 raft_consensus.cc:1261] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db [term 1 FOLLOWER]: Refusing update from remote peer c7322f96571b4b7cb4ce36781fd93525: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:18.718832 28771 consensus_queue.cc:1035] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e38a8b1599499c92e259b76b8efef9" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 33039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:18.719080 28769 consensus_queue.cc:1035] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [LEADER]: Connected to new peer: Peer: permanent_uuid: "282f6ed7531c4d54838790da367ee2db" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 33389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:18.730091 28776 mvcc.cc:205] Tried to move back new op lower bound from 7298937400178683904 to 7298937400086458368. Current Snapshot: MvccSnapshot[applied={T|T < 7298937400178683904}]
I20260620 14:52:18.737308 28284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:18.737553 28284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:18.740356 28284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2559f24ad9d34f788ebb5b61a282e715: Sending DeleteTablet for 3 replicas of tablet 708a481c0830473f958227d585ddf12f
I20260620 14:52:18.741287 28678 tablet_service.cc:1514] Processing DeleteTablet for tablet 708a481c0830473f958227d585ddf12f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:18 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:35322
I20260620 14:52:18.741268 28410 tablet_service.cc:1514] Processing DeleteTablet for tablet 708a481c0830473f958227d585ddf12f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:18 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:56044
I20260620 14:52:18.741350 28544 tablet_service.cc:1514] Processing DeleteTablet for tablet 708a481c0830473f958227d585ddf12f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:18 UTC) from {username='kudu', principal='kudu/127.8.221.254@KRBTEST.COM'} at 127.0.0.1:46664
I20260620 14:52:18.742002 28786 tablet_replica.cc:330] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db: stopping tablet replica
I20260620 14:52:18.742347 28786 raft_consensus.cc:2229] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:18.742355 28785 tablet_replica.cc:330] T 708a481c0830473f958227d585ddf12f P f8e38a8b1599499c92e259b76b8efef9: stopping tablet replica
I20260620 14:52:18.742676 28786 raft_consensus.cc:2258] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:18.742704 28785 raft_consensus.cc:2229] T 708a481c0830473f958227d585ddf12f P f8e38a8b1599499c92e259b76b8efef9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:18.742823 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 28347
I20260620 14:52:18.743067 28785 raft_consensus.cc:2258] T 708a481c0830473f958227d585ddf12f P f8e38a8b1599499c92e259b76b8efef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:18.743620 28784 tablet_replica.cc:330] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525: stopping tablet replica
I20260620 14:52:18.743873 28784 raft_consensus.cc:2229] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:52:18.744236 28784 raft_consensus.cc:2258] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:18.747246 28786 ts_tablet_manager.cc:1905] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:18.750098 28786 ts_tablet_manager.cc:1918] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:18.750231 28786 log.cc:1196] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/708a481c0830473f958227d585ddf12f
I20260620 14:52:18.750622 28786 ts_tablet_manager.cc:1939] T 708a481c0830473f958227d585ddf12f P 282f6ed7531c4d54838790da367ee2db: Deleting consensus metadata
W20260620 14:52:18.751744 28269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.221.193:33039 (error 108)
I20260620 14:52:18.751988 28784 ts_tablet_manager.cc:1905] T 708a481c0830473f958227d585ddf12f P c7322f96571b4b7cb4ce36781fd93525: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:18.752266 28270 catalog_manager.cc:4841] TS 282f6ed7531c4d54838790da367ee2db (127.8.221.195:33389): tablet 708a481c0830473f958227d585ddf12f (table test-table [id=c1651b6015444623b20f686890080a04]) successfully deleted
I20260620 14:52:18.752985 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 28481
I20260620 14:52:18.753940 28757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.753064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:33039 (local address 127.0.0.1:56054)
0620 14:52:18.753699 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:18.753817 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.193:33039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519}
W20260620 14:52:18.754201 28269 catalog_manager.cc:4584] TS f8e38a8b1599499c92e259b76b8efef9 (127.8.221.193:33039): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 708a481c0830473f958227d585ddf12f: Network error: Client connection negotiation failed: client connection to 127.8.221.193:33039: connect: Connection refused (error 111)
I20260620 14:52:18.760738 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 28615
I20260620 14:52:18.761679 28757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.760820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:44195 (local address 127.0.0.1:46668)
0620 14:52:18.761523 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:18.761584 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.194:44195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":599}
I20260620 14:52:18.767167 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 28253
2026-06-20T14:52:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2543 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T14:52:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:18Z Disabled control of system clock
I20260620 14:52:18.802033 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:33953
--webserver_interface=127.8.221.254
--webserver_port=0
--builtin_ntp_servers=127.8.221.212:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:33953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:52:18.918201 28791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:18.918483 28791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:18.918517 28791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:18.922581 28791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:18.922649 28791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:18.922664 28791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:18.922681 28791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:18.922703 28791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:18.927488 28791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:33953
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:33953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:18.928596 28791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:18.930008 28791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:18.939368 28799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:18.939368 28797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:18.939386 28796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:18.939781 28791 server_base.cc:1034] running on GCE node
I20260620 14:52:18.940402 28791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:18.941536 28791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:18.942730 28791 hybrid_clock.cc:648] HybridClock initialized: now 1781967138942736 us; error 101 us; skew 500 ppm
I20260620 14:52:18.947898 28791 webserver.cc:460] Webserver started at http://127.8.221.254:39237/ using document root <none> and password file <none>
I20260620 14:52:18.948544 28791 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:18.948633 28791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:18.948825 28791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:18.951185 28791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "992c03fb7ba24ac489decfb8815b54c0"
format_stamp: "Formatted at 2026-06-20 14:52:18 on dist-test-slave-65tz"
server_key: "e243f067aae1a6ea0bf95c91b66c9112"
server_key_iv: "78291c8772e784baa22fea911275eb0a"
server_key_version: "encryptionkey@0"
I20260620 14:52:18.951743 28791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "992c03fb7ba24ac489decfb8815b54c0"
format_stamp: "Formatted at 2026-06-20 14:52:18 on dist-test-slave-65tz"
server_key: "e243f067aae1a6ea0bf95c91b66c9112"
server_key_iv: "78291c8772e784baa22fea911275eb0a"
server_key_version: "encryptionkey@0"
I20260620 14:52:18.956125 28791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:52:18.959136 28805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:18.960733 28791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:18.960878 28791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "992c03fb7ba24ac489decfb8815b54c0"
format_stamp: "Formatted at 2026-06-20 14:52:18 on dist-test-slave-65tz"
server_key: "e243f067aae1a6ea0bf95c91b66c9112"
server_key_iv: "78291c8772e784baa22fea911275eb0a"
server_key_version: "encryptionkey@0"
I20260620 14:52:18.960987 28791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:18.978474 28791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:18.979183 28791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:18.979351 28791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:18.987367 28791 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:33953
I20260620 14:52:18.987516 28857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:33953 every 8 connection(s)
I20260620 14:52:18.988448 28791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:18.989312 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 28791
I20260620 14:52:18.989500 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:18.989905 9079 external_mini_cluster.cc:1428] Setting key c869da4d80cb8cc021d376bb9c46bb38
I20260620 14:52:18.993119 28858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:18.999838 28858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0: Bootstrap starting.
I20260620 14:52:18.999912 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.991670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39288 (local address 127.8.221.254:33953)
0620 14:52:18.992510 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.992528 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.992567 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:52:18.993308 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.993327 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.993772 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.994183 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:18.994630 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.995614 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.996502 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.996997 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.997178 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.998110 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:18.998115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:18.998127 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:18.998529 (+ 402us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:18.998586 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:18.998606 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:18.998729 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.998841 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.999154 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":193,"threads_started":1}
I20260620 14:52:19.003067 28858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:19.003981 28858 log.cc:824] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:19.006779 28858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0: No bootstrap required, opened a new log
I20260620 14:52:19.011361 28858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992c03fb7ba24ac489decfb8815b54c0" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33953 } }
I20260620 14:52:19.011677 28858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:19.011770 28858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 992c03fb7ba24ac489decfb8815b54c0, State: Initialized, Role: FOLLOWER
I20260620 14:52:19.012288 28858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992c03fb7ba24ac489decfb8815b54c0" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33953 } }
I20260620 14:52:19.012458 28858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:19.012549 28858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:19.012678 28858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:19.013935 28858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992c03fb7ba24ac489decfb8815b54c0" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33953 } }
I20260620 14:52:19.014345 28858 leader_election.cc:302] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 992c03fb7ba24ac489decfb8815b54c0; no voters:
I20260620 14:52:19.014719 28858 leader_election.cc:288] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:19.014899 28863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:19.015195 28863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0 [term 1 LEADER]: Becoming Leader. State: Replica: 992c03fb7ba24ac489decfb8815b54c0, State: Running, Role: LEADER
I20260620 14:52:19.015635 28863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992c03fb7ba24ac489decfb8815b54c0" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33953 } }
I20260620 14:52:19.016358 28858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:19.017580 28864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "992c03fb7ba24ac489decfb8815b54c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992c03fb7ba24ac489decfb8815b54c0" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33953 } } }
I20260620 14:52:19.017768 28864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:19.017580 28865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 992c03fb7ba24ac489decfb8815b54c0. Latest consensus state: current_term: 1 leader_uuid: "992c03fb7ba24ac489decfb8815b54c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "992c03fb7ba24ac489decfb8815b54c0" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33953 } } }
I20260620 14:52:19.018060 28865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:19.018316 28872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:19.021869 28872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:19.029599 28872 catalog_manager.cc:1347] Generated new cluster ID: 7ed6565234694bef94f05266dcdd1a67
I20260620 14:52:19.029685 28872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:19.042220 28872 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:19.043286 28872 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:19.056491 28872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0: Generated new TSK 0
I20260620 14:52:19.057186 28872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:19.068205 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.193:0
--local_ip_for_outbound_sockets=127.8.221.193
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33953
--builtin_ntp_servers=127.8.221.212:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:52:19.196316 28882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:19.196587 28882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:19.196635 28882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:19.200611 28882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:19.200671 28882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:19.200727 28882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.193
I20260620 14:52:19.205562 28882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.221.193
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.193
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:19.206682 28882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:19.207995 28882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:19.218449 28888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:19.218493 28890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:19.218449 28887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:19.218746 28882 server_base.cc:1034] running on GCE node
I20260620 14:52:19.219339 28882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:19.220001 28882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:19.221211 28882 hybrid_clock.cc:648] HybridClock initialized: now 1781967139221192 us; error 37 us; skew 500 ppm
I20260620 14:52:19.223291 28882 webserver.cc:460] Webserver started at http://127.8.221.193:42149/ using document root <none> and password file <none>
I20260620 14:52:19.223896 28882 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:19.223992 28882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:19.224236 28882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:19.226428 28882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-0/data/instance:
uuid: "0959c83061974a48834abf0f6ac35128"
format_stamp: "Formatted at 2026-06-20 14:52:19 on dist-test-slave-65tz"
server_key: "777ff5231e47dff7f7075c21d5a6a002"
server_key_iv: "dfac1847b6524f76b06fba8b0f3a2441"
server_key_version: "encryptionkey@0"
I20260620 14:52:19.226975 28882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance:
uuid: "0959c83061974a48834abf0f6ac35128"
format_stamp: "Formatted at 2026-06-20 14:52:19 on dist-test-slave-65tz"
server_key: "777ff5231e47dff7f7075c21d5a6a002"
server_key_iv: "dfac1847b6524f76b06fba8b0f3a2441"
server_key_version: "encryptionkey@0"
I20260620 14:52:19.231087 28882 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:52:19.234148 28896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:19.235510 28882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:52:19.235653 28882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-0/wal
uuid: "0959c83061974a48834abf0f6ac35128"
format_stamp: "Formatted at 2026-06-20 14:52:19 on dist-test-slave-65tz"
server_key: "777ff5231e47dff7f7075c21d5a6a002"
server_key_iv: "dfac1847b6524f76b06fba8b0f3a2441"
server_key_version: "encryptionkey@0"
I20260620 14:52:19.235795 28882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:19.268247 28882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:19.269411 28882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:19.269623 28882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:19.270367 28882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:19.271435 28882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:19.271515 28882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:19.271585 28882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:19.271620 28882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:19.282691 28882 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.193:34897
I20260620 14:52:19.282814 29009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.193:34897 every 8 connection(s)
I20260620 14:52:19.283717 28882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-0/data/info.pb
I20260620 14:52:19.287081 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 28882
I20260620 14:52:19.287261 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-0/wal/instance
I20260620 14:52:19.287647 9079 external_mini_cluster.cc:1428] Setting key 5d55df09346df5dddd2d760bff8c8a28
I20260620 14:52:19.290369 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.194:0
--local_ip_for_outbound_sockets=127.8.221.194
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33953
--builtin_ntp_servers=127.8.221.212:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:52:19.296016 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.286150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.193:35883 (local address 127.8.221.254:33953)
0620 14:52:19.286315 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:19.286319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:19.287890 (+ 1571us) server_negotiation.cc:407] Connection header received
0620 14:52:19.289935 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:19.289938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:19.289988 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:19.290057 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:19.291618 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.292374 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.293716 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.294017 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.294132 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:19.295472 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:19.295476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:19.295478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:19.295491 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:19.295509 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:19.295517 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:19.295596 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:19.295771 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:52:19.295858 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:52:19.296962 29012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.286590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:33953 (local address 127.8.221.193:35883)
0620 14:52:19.287629 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:19.287708 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:19.289635 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:19.290303 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:19.290314 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:19.290468 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:19.291382 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:19.291406 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.292555 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.292559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:19.293564 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:19.293577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.294183 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.294187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:19.294367 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:19.295173 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:52:19.295175 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:19.295218 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:52:19.295293 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:19.295598 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:19.295609 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:19.295616 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:52:19.295761 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 14:52:19.296064 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":139,"threads_started":1}
I20260620 14:52:19.298616 29010 heartbeater.cc:347] Connected to a master server at 127.8.221.254:33953
I20260620 14:52:19.298883 29010 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:19.299428 29010 heartbeater.cc:511] Master 127.8.221.254:33953 requested a full tablet report, sending...
I20260620 14:52:19.300957 28822 ts_manager.cc:194] Registered new tserver with Master: 0959c83061974a48834abf0f6ac35128 (127.8.221.193:34897)
I20260620 14:52:19.302186 28822 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.221.193:35883
W20260620 14:52:19.429179 29013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:19.429708 29013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:19.429818 29013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:19.436538 29013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:19.436651 29013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:19.436762 29013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.194
I20260620 14:52:19.444623 29013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.221.194
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.194
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:19.446226 29013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:19.448055 29013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:19.459635 29018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:19.459690 29021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:19.459834 29013 server_base.cc:1034] running on GCE node
W20260620 14:52:19.459740 29019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:19.460436 29013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:19.461220 29013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:19.462391 29013 hybrid_clock.cc:648] HybridClock initialized: now 1781967139462371 us; error 37 us; skew 500 ppm
I20260620 14:52:19.464638 29013 webserver.cc:460] Webserver started at http://127.8.221.194:36389/ using document root <none> and password file <none>
I20260620 14:52:19.465385 29013 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:19.465454 29013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:19.465713 29013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:19.468304 29013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-1/data/instance:
uuid: "d812b2ac57d34c5c95429edb1060c9e9"
format_stamp: "Formatted at 2026-06-20 14:52:19 on dist-test-slave-65tz"
server_key: "f6251e4cd747a418951885e117ae0a37"
server_key_iv: "4d03b75eba4f6e583fd510751af6ff32"
server_key_version: "encryptionkey@0"
I20260620 14:52:19.469194 29013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance:
uuid: "d812b2ac57d34c5c95429edb1060c9e9"
format_stamp: "Formatted at 2026-06-20 14:52:19 on dist-test-slave-65tz"
server_key: "f6251e4cd747a418951885e117ae0a37"
server_key_iv: "4d03b75eba4f6e583fd510751af6ff32"
server_key_version: "encryptionkey@0"
I20260620 14:52:19.474875 29013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:52:19.478184 29027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:19.479728 29013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:52:19.479892 29013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-1/wal
uuid: "d812b2ac57d34c5c95429edb1060c9e9"
format_stamp: "Formatted at 2026-06-20 14:52:19 on dist-test-slave-65tz"
server_key: "f6251e4cd747a418951885e117ae0a37"
server_key_iv: "4d03b75eba4f6e583fd510751af6ff32"
server_key_version: "encryptionkey@0"
I20260620 14:52:19.480072 29013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:19.494812 29013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:19.495633 29013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:19.495854 29013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:19.496547 29013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:19.497645 29013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:19.497721 29013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:19.497843 29013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:19.497898 29013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 14:52:19.509527 29013 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.194:45431
I20260620 14:52:19.509550 29140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.194:45431 every 8 connection(s)
I20260620 14:52:19.510828 29013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-1/data/info.pb
I20260620 14:52:19.519078 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 29013
I20260620 14:52:19.519219 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-1/wal/instance
I20260620 14:52:19.519541 9079 external_mini_cluster.cc:1428] Setting key dc0f3466fd6d8e32bf32afcb3d84201d
I20260620 14:52:19.522032 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.221.195:0
--local_ip_for_outbound_sockets=127.8.221.195
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33953
--builtin_ntp_servers=127.8.221.212:39353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:52:19.522343 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.513102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.194:35365 (local address 127.8.221.254:33953)
0620 14:52:19.513284 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:19.513289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:19.514700 (+ 1411us) server_negotiation.cc:407] Connection header received
0620 14:52:19.516237 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:19.516242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:19.516341 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:19.516438 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:19.517607 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.518243 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.519749 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.520099 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.520242 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:19.521674 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:19.521679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:19.521684 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:19.521701 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:19.521731 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:19.521745 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:19.521885 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:19.522040 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:52:19.522127 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:52:19.523283 29143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.513580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:33953 (local address 127.8.221.194:35365)
0620 14:52:19.514510 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:19.514563 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:19.515950 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:19.516617 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:19.516623 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:19.516708 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:19.517396 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:19.517419 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.518373 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.518377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:19.519566 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:19.519579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.520512 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.520516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:19.520703 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:19.521402 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:52:19.521403 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:19.521444 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:52:19.521508 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:19.521883 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:19.521890 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:19.521895 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:52:19.522006 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:52:19.522358 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":208,"threads_started":1}
I20260620 14:52:19.524507 29141 heartbeater.cc:347] Connected to a master server at 127.8.221.254:33953
I20260620 14:52:19.524737 29141 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:19.525276 29141 heartbeater.cc:511] Master 127.8.221.254:33953 requested a full tablet report, sending...
I20260620 14:52:19.526330 28822 ts_manager.cc:194] Registered new tserver with Master: d812b2ac57d34c5c95429edb1060c9e9 (127.8.221.194:45431)
I20260620 14:52:19.526924 28822 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.221.194:35365
W20260620 14:52:19.652077 29144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:19.652413 29144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:19.652486 29144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:19.656736 29144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:19.656832 29144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:19.656929 29144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.221.195
I20260620 14:52:19.662398 29144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:39353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.221.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.221.195
--webserver_port=0
--tserver_master_addrs=127.8.221.254:33953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.221.195
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:19.663915 29144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:19.665876 29144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:19.677908 29152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:19.677908 29150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:19.678010 29149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:19.678297 29144 server_base.cc:1034] running on GCE node
I20260620 14:52:19.678823 29144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:19.679737 29144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:19.680936 29144 hybrid_clock.cc:648] HybridClock initialized: now 1781967139680893 us; error 59 us; skew 500 ppm
I20260620 14:52:19.683377 29144 webserver.cc:460] Webserver started at http://127.8.221.195:38185/ using document root <none> and password file <none>
I20260620 14:52:19.684159 29144 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:19.684258 29144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:19.684525 29144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:19.687110 29144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-2/data/instance:
uuid: "3dbc34f70eb9449797e3bd46d8c00934"
format_stamp: "Formatted at 2026-06-20 14:52:19 on dist-test-slave-65tz"
server_key: "84d294af80062eb479fd82ef72281806"
server_key_iv: "9beb6cb564523c081e1fc8cb1e89efe8"
server_key_version: "encryptionkey@0"
I20260620 14:52:19.687744 29144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance:
uuid: "3dbc34f70eb9449797e3bd46d8c00934"
format_stamp: "Formatted at 2026-06-20 14:52:19 on dist-test-slave-65tz"
server_key: "84d294af80062eb479fd82ef72281806"
server_key_iv: "9beb6cb564523c081e1fc8cb1e89efe8"
server_key_version: "encryptionkey@0"
I20260620 14:52:19.692507 29144 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:52:19.695858 29158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:19.697918 29144 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:52:19.698112 29144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-2/wal
uuid: "3dbc34f70eb9449797e3bd46d8c00934"
format_stamp: "Formatted at 2026-06-20 14:52:19 on dist-test-slave-65tz"
server_key: "84d294af80062eb479fd82ef72281806"
server_key_iv: "9beb6cb564523c081e1fc8cb1e89efe8"
server_key_version: "encryptionkey@0"
I20260620 14:52:19.698304 29144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:19.714797 29144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:19.715624 29144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:19.715797 29144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:19.716516 29144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:19.717712 29144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:19.717787 29144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:19.717892 29144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:19.717940 29144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:19.729176 29144 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.195:34873
I20260620 14:52:19.729218 29271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.195:34873 every 8 connection(s)
I20260620 14:52:19.730455 29144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-2/data/info.pb
I20260620 14:52:19.740638 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 29144
I20260620 14:52:19.740824 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-2/wal/instance
I20260620 14:52:19.741252 9079 external_mini_cluster.cc:1428] Setting key aef8be85aa2c049e53d7a8c55802322c
I20260620 14:52:19.741345 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.732811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:53745 (local address 127.8.221.254:33953)
0620 14:52:19.732961 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:19.732965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:19.734381 (+ 1416us) server_negotiation.cc:407] Connection header received
0620 14:52:19.735818 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:19.735821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:19.735884 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:19.735986 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:19.737046 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.737622 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.738870 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.739101 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.739192 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:19.740699 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:19.740705 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:19.740712 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:19.740732 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:19.740764 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:19.740777 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:19.740874 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:19.741076 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 14:52:19.741163 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:52:19.742383 29274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.733314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:33953 (local address 127.8.221.195:53745)
0620 14:52:19.734190 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:19.734236 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:19.735600 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:19.736177 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:19.736184 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:19.736306 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:19.736907 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:19.736920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.737735 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.737737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:19.738746 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:19.738755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.739348 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.739356 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:19.739527 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:19.740218 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:52:19.740219 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:19.740260 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:52:19.740327 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:19.740881 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:19.740893 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:19.740910 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:52:19.741041 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 14:52:19.741278 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":143,"threads_started":1}
I20260620 14:52:19.744109 29272 heartbeater.cc:347] Connected to a master server at 127.8.221.254:33953
I20260620 14:52:19.744395 29272 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:19.744995 29272 heartbeater.cc:511] Master 127.8.221.254:33953 requested a full tablet report, sending...
I20260620 14:52:19.746141 28822 ts_manager.cc:194] Registered new tserver with Master: 3dbc34f70eb9449797e3bd46d8c00934 (127.8.221.195:34873)
I20260620 14:52:19.746713 28822 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.221.195:53745
I20260620 14:52:19.756379 9079 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:52:19.763062 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.758590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39300 (local address 127.8.221.254:33953)
0620 14:52:19.758766 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:19.758771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:19.759014 (+ 243us) server_negotiation.cc:407] Connection header received
0620 14:52:19.759165 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:19.759167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:19.759248 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:19.759327 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:19.759677 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.760221 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.761211 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.761420 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.761500 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:19.762576 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:19.762579 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:19.762581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:19.762594 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:19.762616 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:19.762624 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:19.762699 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:19.762806 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 14:52:19.762857 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:52:19.766614 28822 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:52:19.769690 28822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:52:19.787202 29286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.780193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:45431 (local address 127.0.0.1:36244)
0620 14:52:19.781222 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:19.781239 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:19.781372 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:19.782100 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:19.782103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:19.782144 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:19.782281 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:19.782289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.784026 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.784031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:19.785398 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:19.785407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.786507 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.786511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:19.786630 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:19.786687 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 14:52:19.786830 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 14:52:19.786996 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":79,"thread_start_us":137,"threads_started":1}
I20260620 14:52:19.787567 29285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.780411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.195:34873 (local address 127.0.0.1:33626)
0620 14:52:19.781480 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:19.781495 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:19.781577 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:19.782214 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:19.782217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:19.782233 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:19.782320 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:19.782325 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.783578 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.783581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:19.785495 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:19.785501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.787036 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.787040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:19.787148 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:19.787181 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:52:19.787255 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:52:19.787367 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":258,"thread_start_us":137,"threads_started":1}
I20260620 14:52:19.787209 29283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.780020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:34897 (local address 127.0.0.1:48980)
0620 14:52:19.780937 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:19.781002 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:19.781221 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:19.781848 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:19.781855 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:19.781881 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:19.782053 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:19.782062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.783695 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.783700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:19.785477 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:19.785490 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.786883 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.786887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:19.786981 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:19.787003 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:52:19.787054 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 14:52:19.787120 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":191,"thread_start_us":92,"threads_started":1}
I20260620 14:52:19.787907 29287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.780935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36244 (local address 127.8.221.194:45431)
0620 14:52:19.781541 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:19.781551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:19.781572 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:52:19.781640 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:19.781644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:19.781832 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:19.781959 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:19.782423 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.783909 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.785515 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.786342 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.786461 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:19.786677 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:19.787535 (+ 858us) server_negotiation.cc:299] Negotiation successful
0620 14:52:19.787703 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":149,"threads_started":1}
I20260620 14:52:19.788175 29288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.781041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33626 (local address 127.8.221.195:34873)
0620 14:52:19.781671 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:19.781696 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:19.781719 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:52:19.781852 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:19.781857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:19.782084 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:19.782247 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:19.782458 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.783450 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.785626 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.786469 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.786739 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:19.786923 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:19.787859 (+ 936us) server_negotiation.cc:299] Negotiation successful
0620 14:52:19.787975 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":113,"threads_started":1}
I20260620 14:52:19.788511 29284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.780198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48980 (local address 127.8.221.193:34897)
0620 14:52:19.780670 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:19.780678 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:19.781101 (+ 423us) server_negotiation.cc:407] Connection header received
0620 14:52:19.781411 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:19.781418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:19.781647 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:19.781840 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:19.782189 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.783575 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.785981 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.786741 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.787895 (+ 1154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:19.788169 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:19.788214 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 14:52:19.788363 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":142,"threads_started":1}
I20260620 14:52:19.791811 28944 tablet_service.cc:1467] Processing CreateTablet for tablet 142401bba4e14146ad719e4aaa6385fe (DEFAULT_TABLE table=test-table [id=4b6b7d5e8a5745a3b04561170dacce17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:19.791882 29075 tablet_service.cc:1467] Processing CreateTablet for tablet 142401bba4e14146ad719e4aaa6385fe (DEFAULT_TABLE table=test-table [id=4b6b7d5e8a5745a3b04561170dacce17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:19.791883 29206 tablet_service.cc:1467] Processing CreateTablet for tablet 142401bba4e14146ad719e4aaa6385fe (DEFAULT_TABLE table=test-table [id=4b6b7d5e8a5745a3b04561170dacce17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:19.792850 28944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142401bba4e14146ad719e4aaa6385fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:19.793129 29075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142401bba4e14146ad719e4aaa6385fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:19.793129 29206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 142401bba4e14146ad719e4aaa6385fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:19.799819 29289 tablet_bootstrap.cc:492] T 142401bba4e14146ad719e4aaa6385fe P 0959c83061974a48834abf0f6ac35128: Bootstrap starting.
I20260620 14:52:19.802266 29289 tablet_bootstrap.cc:654] T 142401bba4e14146ad719e4aaa6385fe P 0959c83061974a48834abf0f6ac35128: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:19.802647 29290 tablet_bootstrap.cc:492] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9: Bootstrap starting.
I20260620 14:52:19.803171 29289 log.cc:824] T 142401bba4e14146ad719e4aaa6385fe P 0959c83061974a48834abf0f6ac35128: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:19.805066 29289 tablet_bootstrap.cc:492] T 142401bba4e14146ad719e4aaa6385fe P 0959c83061974a48834abf0f6ac35128: No bootstrap required, opened a new log
I20260620 14:52:19.805271 29289 ts_tablet_manager.cc:1397] T 142401bba4e14146ad719e4aaa6385fe P 0959c83061974a48834abf0f6ac35128: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 14:52:19.805292 29290 tablet_bootstrap.cc:654] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:19.806278 29291 tablet_bootstrap.cc:492] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934: Bootstrap starting.
I20260620 14:52:19.806385 29290 log.cc:824] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:19.808980 29291 tablet_bootstrap.cc:654] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:19.809126 29290 tablet_bootstrap.cc:492] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9: No bootstrap required, opened a new log
I20260620 14:52:19.809381 29290 ts_tablet_manager.cc:1397] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 14:52:19.809299 29289 raft_consensus.cc:348] T 142401bba4e14146ad719e4aaa6385fe P 0959c83061974a48834abf0f6ac35128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0959c83061974a48834abf0f6ac35128" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 34897 } } peers { permanent_uuid: "d812b2ac57d34c5c95429edb1060c9e9" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45431 } } peers { permanent_uuid: "3dbc34f70eb9449797e3bd46d8c00934" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34873 } }
I20260620 14:52:19.809590 29289 raft_consensus.cc:374] T 142401bba4e14146ad719e4aaa6385fe P 0959c83061974a48834abf0f6ac35128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:19.809651 29289 raft_consensus.cc:729] T 142401bba4e14146ad719e4aaa6385fe P 0959c83061974a48834abf0f6ac35128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0959c83061974a48834abf0f6ac35128, State: Initialized, Role: FOLLOWER
I20260620 14:52:19.810132 29291 log.cc:824] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:19.810166 29289 consensus_queue.cc:260] T 142401bba4e14146ad719e4aaa6385fe P 0959c83061974a48834abf0f6ac35128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0959c83061974a48834abf0f6ac35128" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 34897 } } peers { permanent_uuid: "d812b2ac57d34c5c95429edb1060c9e9" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45431 } } peers { permanent_uuid: "3dbc34f70eb9449797e3bd46d8c00934" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34873 } }
I20260620 14:52:19.810873 29010 heartbeater.cc:503] Master 127.8.221.254:33953 was elected leader, sending a full tablet report...
I20260620 14:52:19.812219 29289 ts_tablet_manager.cc:1428] T 142401bba4e14146ad719e4aaa6385fe P 0959c83061974a48834abf0f6ac35128: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 14:52:19.814440 29290 raft_consensus.cc:348] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0959c83061974a48834abf0f6ac35128" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 34897 } } peers { permanent_uuid: "d812b2ac57d34c5c95429edb1060c9e9" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45431 } } peers { permanent_uuid: "3dbc34f70eb9449797e3bd46d8c00934" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34873 } }
I20260620 14:52:19.814810 29290 raft_consensus.cc:374] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:19.814890 29291 tablet_bootstrap.cc:492] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934: No bootstrap required, opened a new log
I20260620 14:52:19.814881 29290 raft_consensus.cc:729] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d812b2ac57d34c5c95429edb1060c9e9, State: Initialized, Role: FOLLOWER
I20260620 14:52:19.815148 29291 ts_tablet_manager.cc:1397] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 14:52:19.815519 29290 consensus_queue.cc:260] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0959c83061974a48834abf0f6ac35128" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 34897 } } peers { permanent_uuid: "d812b2ac57d34c5c95429edb1060c9e9" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45431 } } peers { permanent_uuid: "3dbc34f70eb9449797e3bd46d8c00934" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34873 } }
I20260620 14:52:19.816416 29141 heartbeater.cc:503] Master 127.8.221.254:33953 was elected leader, sending a full tablet report...
I20260620 14:52:19.816848 29290 ts_tablet_manager.cc:1428] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20260620 14:52:19.819921 29291 raft_consensus.cc:348] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0959c83061974a48834abf0f6ac35128" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 34897 } } peers { permanent_uuid: "d812b2ac57d34c5c95429edb1060c9e9" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45431 } } peers { permanent_uuid: "3dbc34f70eb9449797e3bd46d8c00934" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34873 } }
I20260620 14:52:19.820222 29291 raft_consensus.cc:374] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:19.820314 29291 raft_consensus.cc:729] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dbc34f70eb9449797e3bd46d8c00934, State: Initialized, Role: FOLLOWER
I20260620 14:52:19.820833 29291 consensus_queue.cc:260] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0959c83061974a48834abf0f6ac35128" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 34897 } } peers { permanent_uuid: "d812b2ac57d34c5c95429edb1060c9e9" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45431 } } peers { permanent_uuid: "3dbc34f70eb9449797e3bd46d8c00934" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34873 } }
I20260620 14:52:19.821599 29272 heartbeater.cc:503] Master 127.8.221.254:33953 was elected leader, sending a full tablet report...
I20260620 14:52:19.821998 29291 ts_tablet_manager.cc:1428] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260620 14:52:19.880218 29297 raft_consensus.cc:482] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:52:19.880569 29297 raft_consensus.cc:504] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0959c83061974a48834abf0f6ac35128" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 34897 } } peers { permanent_uuid: "d812b2ac57d34c5c95429edb1060c9e9" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45431 } } peers { permanent_uuid: "3dbc34f70eb9449797e3bd46d8c00934" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34873 } }
I20260620 14:52:19.882135 29297 leader_election.cc:288] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0959c83061974a48834abf0f6ac35128 (127.8.221.193:34897), d812b2ac57d34c5c95429edb1060c9e9 (127.8.221.194:45431)
I20260620 14:52:19.887135 29287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.882816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:38553 (local address 127.8.221.194:45431)
0620 14:52:19.883016 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:19.883023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:19.883049 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:52:19.883247 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:19.883250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:19.883306 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:19.883457 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:19.883826 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.884555 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.886029 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.886446 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.886587 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:19.886657 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:19.886900 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 14:52:19.886988 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:52:19.887135 29274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.882506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:45431 (local address 127.8.221.195:38553)
0620 14:52:19.882948 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:19.882970 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:19.883091 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:19.883482 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:19.883488 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:19.883507 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:19.883682 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:19.883689 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.884699 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.884702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:19.885894 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:19.885908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.886579 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.886582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:19.886770 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:19.886789 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 14:52:19.886882 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:52:19.886976 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":201}
I20260620 14:52:19.887138 29284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.882875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.221.195:33851 (local address 127.8.221.193:34897)
0620 14:52:19.883043 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:19.883048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:19.883407 (+ 359us) server_negotiation.cc:407] Connection header received
0620 14:52:19.883493 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:19.883496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:19.883556 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:19.883694 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:19.884087 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.884633 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.885990 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.886435 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.886583 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:19.886653 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:19.886882 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 14:52:19.886958 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:52:19.887176 29298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.882742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:34897 (local address 127.8.221.195:33851)
0620 14:52:19.883284 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:19.883296 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:19.883382 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:19.883722 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:19.883726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:19.883754 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:19.883956 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:19.883963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.884761 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.884767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:19.885846 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:19.885857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.886577 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.886581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:19.886742 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:19.886766 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:52:19.886866 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:52:19.886959 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":33,"thread_start_us":181,"threads_started":1}
I20260620 14:52:19.888046 29095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142401bba4e14146ad719e4aaa6385fe" candidate_uuid: "3dbc34f70eb9449797e3bd46d8c00934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d812b2ac57d34c5c95429edb1060c9e9" is_pre_election: true
I20260620 14:52:19.888374 28964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142401bba4e14146ad719e4aaa6385fe" candidate_uuid: "3dbc34f70eb9449797e3bd46d8c00934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0959c83061974a48834abf0f6ac35128" is_pre_election: true
I20260620 14:52:19.888446 29095 raft_consensus.cc:2454] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dbc34f70eb9449797e3bd46d8c00934 in term 0.
I20260620 14:52:19.888734 28964 raft_consensus.cc:2454] T 142401bba4e14146ad719e4aaa6385fe P 0959c83061974a48834abf0f6ac35128 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dbc34f70eb9449797e3bd46d8c00934 in term 0.
I20260620 14:52:19.889076 29160 leader_election.cc:302] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dbc34f70eb9449797e3bd46d8c00934, d812b2ac57d34c5c95429edb1060c9e9; no voters:
I20260620 14:52:19.889420 29297 raft_consensus.cc:2781] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:52:19.889521 29297 raft_consensus.cc:482] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:52:19.889565 29297 raft_consensus.cc:3037] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:19.890836 29297 raft_consensus.cc:504] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0959c83061974a48834abf0f6ac35128" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 34897 } } peers { permanent_uuid: "d812b2ac57d34c5c95429edb1060c9e9" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45431 } } peers { permanent_uuid: "3dbc34f70eb9449797e3bd46d8c00934" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34873 } }
I20260620 14:52:19.891239 29297 leader_election.cc:288] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [CANDIDATE]: Term 1 election: Requested vote from peers 0959c83061974a48834abf0f6ac35128 (127.8.221.193:34897), d812b2ac57d34c5c95429edb1060c9e9 (127.8.221.194:45431)
I20260620 14:52:19.891700 29095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142401bba4e14146ad719e4aaa6385fe" candidate_uuid: "3dbc34f70eb9449797e3bd46d8c00934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d812b2ac57d34c5c95429edb1060c9e9"
I20260620 14:52:19.891727 28964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "142401bba4e14146ad719e4aaa6385fe" candidate_uuid: "3dbc34f70eb9449797e3bd46d8c00934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0959c83061974a48834abf0f6ac35128"
I20260620 14:52:19.891866 28964 raft_consensus.cc:3037] T 142401bba4e14146ad719e4aaa6385fe P 0959c83061974a48834abf0f6ac35128 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:19.891863 29095 raft_consensus.cc:3037] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:19.893110 29095 raft_consensus.cc:2454] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dbc34f70eb9449797e3bd46d8c00934 in term 1.
I20260620 14:52:19.893116 28964 raft_consensus.cc:2454] T 142401bba4e14146ad719e4aaa6385fe P 0959c83061974a48834abf0f6ac35128 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dbc34f70eb9449797e3bd46d8c00934 in term 1.
I20260620 14:52:19.893493 29159 leader_election.cc:302] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0959c83061974a48834abf0f6ac35128, 3dbc34f70eb9449797e3bd46d8c00934; no voters:
I20260620 14:52:19.893698 29297 raft_consensus.cc:2781] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:19.894032 29297 raft_consensus.cc:686] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [term 1 LEADER]: Becoming Leader. State: Replica: 3dbc34f70eb9449797e3bd46d8c00934, State: Running, Role: LEADER
I20260620 14:52:19.894450 29297 consensus_queue.cc:237] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0959c83061974a48834abf0f6ac35128" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 34897 } } peers { permanent_uuid: "d812b2ac57d34c5c95429edb1060c9e9" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45431 } } peers { permanent_uuid: "3dbc34f70eb9449797e3bd46d8c00934" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34873 } }
I20260620 14:52:19.898514 28822 catalog_manager.cc:5495] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dbc34f70eb9449797e3bd46d8c00934 (127.8.221.195). New cstate: current_term: 1 leader_uuid: "3dbc34f70eb9449797e3bd46d8c00934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0959c83061974a48834abf0f6ac35128" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 34897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d812b2ac57d34c5c95429edb1060c9e9" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dbc34f70eb9449797e3bd46d8c00934" member_type: VOTER last_known_addr { host: "127.8.221.195" port: 34873 } health_report { overall_health: HEALTHY } } }
I20260620 14:52:19.940615 29288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.936833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33638 (local address 127.8.221.195:34873)
0620 14:52:19.937031 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:19.937036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:19.937051 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:52:19.937144 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:19.937147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:19.937211 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:19.937333 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:19.937576 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.938123 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.939044 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.939262 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.939372 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:19.939699 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:19.940201 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:19.940332 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:19.940398 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:52:19.940447 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 14:52:19.948263 29095 raft_consensus.cc:1261] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9 [term 1 FOLLOWER]: Refusing update from remote peer 3dbc34f70eb9449797e3bd46d8c00934: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:19.948263 28964 raft_consensus.cc:1261] T 142401bba4e14146ad719e4aaa6385fe P 0959c83061974a48834abf0f6ac35128 [term 1 FOLLOWER]: Refusing update from remote peer 3dbc34f70eb9449797e3bd46d8c00934: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:19.949330 29299 consensus_queue.cc:1035] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d812b2ac57d34c5c95429edb1060c9e9" member_type: VOTER last_known_addr { host: "127.8.221.194" port: 45431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:19.949657 29297 consensus_queue.cc:1035] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0959c83061974a48834abf0f6ac35128" member_type: VOTER last_known_addr { host: "127.8.221.193" port: 34897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:19.958555 29303 mvcc.cc:205] Tried to move back new op lower bound from 7298937405217566720 to 7298937405010952192. Current Snapshot: MvccSnapshot[applied={T|T < 7298937405217566720}]
I20260620 14:52:19.960399 29305 mvcc.cc:205] Tried to move back new op lower bound from 7298937405217566720 to 7298937405010952192. Current Snapshot: MvccSnapshot[applied={T|T < 7298937405217566720}]
I20260620 14:52:19.968607 28821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:19.968832 28821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:19.971817 28821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 992c03fb7ba24ac489decfb8815b54c0: Sending DeleteTablet for 3 replicas of tablet 142401bba4e14146ad719e4aaa6385fe
I20260620 14:52:19.972834 28944 tablet_service.cc:1514] Processing DeleteTablet for tablet 142401bba4e14146ad719e4aaa6385fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:19 UTC) from {username='slave'} at 127.0.0.1:48980
I20260620 14:52:19.972848 29075 tablet_service.cc:1514] Processing DeleteTablet for tablet 142401bba4e14146ad719e4aaa6385fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:19 UTC) from {username='slave'} at 127.0.0.1:36244
I20260620 14:52:19.973264 29206 tablet_service.cc:1514] Processing DeleteTablet for tablet 142401bba4e14146ad719e4aaa6385fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:19 UTC) from {username='slave'} at 127.0.0.1:33626
I20260620 14:52:19.973480 29313 tablet_replica.cc:330] T 142401bba4e14146ad719e4aaa6385fe P 0959c83061974a48834abf0f6ac35128: stopping tablet replica
I20260620 14:52:19.973950 29315 tablet_replica.cc:330] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934: stopping tablet replica
I20260620 14:52:19.973961 29313 raft_consensus.cc:2229] T 142401bba4e14146ad719e4aaa6385fe P 0959c83061974a48834abf0f6ac35128 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:19.974154 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 28882
I20260620 14:52:19.974177 29314 tablet_replica.cc:330] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9: stopping tablet replica
I20260620 14:52:19.976848 29314 raft_consensus.cc:2229] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:19.977238 29314 raft_consensus.cc:2258] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:19.977768 29315 raft_consensus.cc:2229] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:52:19.978228 29315 raft_consensus.cc:2258] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:19.979216 29314 ts_tablet_manager.cc:1905] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:19.980429 29315 ts_tablet_manager.cc:1905] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:19.982954 29314 ts_tablet_manager.cc:1918] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260620 14:52:19.982985 28806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.221.193:34897 (error 108)
I20260620 14:52:19.983072 29314 log.cc:1196] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-1/wal/wals/142401bba4e14146ad719e4aaa6385fe
I20260620 14:52:19.983403 29314 ts_tablet_manager.cc:1939] T 142401bba4e14146ad719e4aaa6385fe P d812b2ac57d34c5c95429edb1060c9e9: Deleting consensus metadata
I20260620 14:52:19.983893 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 29013
I20260620 14:52:19.984068 29315 ts_tablet_manager.cc:1918] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:19.984217 29315 log.cc:1196] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934: Deleting WAL directory at /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966871461349-9079-0/minicluster-data/ts-2/wal/wals/142401bba4e14146ad719e4aaa6385fe
I20260620 14:52:19.984680 29315 ts_tablet_manager.cc:1939] T 142401bba4e14146ad719e4aaa6385fe P 3dbc34f70eb9449797e3bd46d8c00934: Deleting consensus metadata
I20260620 14:52:19.985925 28808 catalog_manager.cc:4841] TS 3dbc34f70eb9449797e3bd46d8c00934 (127.8.221.195:34873): tablet 142401bba4e14146ad719e4aaa6385fe (table test-table [id=4b6b7d5e8a5745a3b04561170dacce17]) successfully deleted
I20260620 14:52:19.990978 29285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.990566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.193:34897 (local address 127.0.0.1:48984)
0620 14:52:19.990805 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:19.990878 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.193:34897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260620 14:52:19.991165 28806 catalog_manager.cc:4584] TS 0959c83061974a48834abf0f6ac35128 (127.8.221.193:34897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 142401bba4e14146ad719e4aaa6385fe: Network error: Client connection negotiation failed: client connection to 127.8.221.193:34897: connect: Connection refused (error 111)
I20260620 14:52:19.993728 29285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.993440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.194:45431 (local address 127.0.0.1:36260)
0620 14:52:19.993590 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:19.993649 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.194:45431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 14:52:19.993880 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 29144
I20260620 14:52:20.002501 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 28791
2026-06-20T14:52:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1232 ms)
[----------] 6 tests from EncryptionPolicyTest (11833 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:20 dist-test-slave-65tz krb5kdc[29319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:20 dist-test-slave-65tz krb5kdc[29319](info): set up 2 sockets
Jun 20 14:52:20 dist-test-slave-65tz krb5kdc[29319](info): commencing operation
krb5kdc: starting...
W20260620 14:52:21.062534 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:21 dist-test-slave-65tz krb5kdc[29319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:21.198107 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42763
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:21.325699 29332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:21.326097 29332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:21.326179 29332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:21.330367 29332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:21.330461 29332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:21.330498 29332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:21.330556 29332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:21.330636 29332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:21.336194 29332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42763
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:21.337514 29332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:21.338840 29332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:21.349390 29337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:21.349409 29338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:21.349409 29340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:21.349772 29332 server_base.cc:1034] running on GCE node
I20260620 14:52:21.350605 29332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:21.351971 29332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:21.353192 29332 hybrid_clock.cc:648] HybridClock initialized: now 1781967141353163 us; error 49 us; skew 500 ppm
Jun 20 14:52:21 dist-test-slave-65tz krb5kdc[29319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:21.356262 29332 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:21.357225 29332 webserver.cc:460] Webserver started at http://127.0.0.1:37271/ using document root <none> and password file <none>
I20260620 14:52:21.357864 29332 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:21.357934 29332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:21.358115 29332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:21.360173 29332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "75a27c589ac548ca8d89bc92a879f8a7"
format_stamp: "Formatted at 2026-06-20 14:52:21 on dist-test-slave-65tz"
server_key: "5f9118080a70fd4f295471002a5cfc22"
server_key_iv: "a7c4b08f3fedaef9f055e5c30600667e"
server_key_version: "encryptionkey@0"
I20260620 14:52:21.360688 29332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "75a27c589ac548ca8d89bc92a879f8a7"
format_stamp: "Formatted at 2026-06-20 14:52:21 on dist-test-slave-65tz"
server_key: "5f9118080a70fd4f295471002a5cfc22"
server_key_iv: "a7c4b08f3fedaef9f055e5c30600667e"
server_key_version: "encryptionkey@0"
I20260620 14:52:21.365094 29332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:52:21.368300 29347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:21.369900 29332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:21.370077 29332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "75a27c589ac548ca8d89bc92a879f8a7"
format_stamp: "Formatted at 2026-06-20 14:52:21 on dist-test-slave-65tz"
server_key: "5f9118080a70fd4f295471002a5cfc22"
server_key_iv: "a7c4b08f3fedaef9f055e5c30600667e"
server_key_version: "encryptionkey@0"
I20260620 14:52:21.370221 29332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:21.388618 29332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:21.392683 29332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:21.392935 29332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:21.401852 29332 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42763
I20260620 14:52:21.401880 29399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42763 every 8 connection(s)
I20260620 14:52:21.403036 29332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:21.405820 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 29332
I20260620 14:52:21.405990 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:21.406343 9079 external_mini_cluster.cc:1428] Setting key 75bb3222205ad765037e5b2a0076d608
I20260620 14:52:21.406363 29400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:21.413556 29400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7: Bootstrap starting.
I20260620 14:52:21.416340 29400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:21.417129 29400 log.cc:824] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:52:21 dist-test-slave-65tz krb5kdc[29319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:52:21.419050 29400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7: No bootstrap required, opened a new log
I20260620 14:52:21.422652 29400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a27c589ac548ca8d89bc92a879f8a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42763 } }
I20260620 14:52:21.422906 29400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:21.422999 29400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a27c589ac548ca8d89bc92a879f8a7, State: Initialized, Role: FOLLOWER
I20260620 14:52:21.423293 29403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:21.408222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50785 (local address 127.0.0.1:42763)
0620 14:52:21.408788 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:21.408807 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:21.408852 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:52:21.409778 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:21.409876 (+ 98us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:21.410320 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:21.410673 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:21.411747 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.412932 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.414158 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.414705 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.414857 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:21.417959 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:21.417964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:21.417981 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:21.418019 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:21.419907 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:21.420488 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:21.420493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:21.420499 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:21.420613 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:21.420956 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:21.420961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:21.420969 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:21.421415 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:21.421674 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:21.421980 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 14:52:21.422258 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":186,"threads_started":1}
I20260620 14:52:21.423523 29400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a27c589ac548ca8d89bc92a879f8a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42763 } }
I20260620 14:52:21.423683 29400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:21.423754 29400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:21.423861 29400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:21.424875 29400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a27c589ac548ca8d89bc92a879f8a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42763 } }
I20260620 14:52:21.425243 29400 leader_election.cc:302] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75a27c589ac548ca8d89bc92a879f8a7; no voters:
I20260620 14:52:21.425580 29400 leader_election.cc:288] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:21.425865 29405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:21.426213 29405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7 [term 1 LEADER]: Becoming Leader. State: Replica: 75a27c589ac548ca8d89bc92a879f8a7, State: Running, Role: LEADER
I20260620 14:52:21.426688 29405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a27c589ac548ca8d89bc92a879f8a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42763 } }
I20260620 14:52:21.426889 29400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:21.427099 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:21.428790 29406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75a27c589ac548ca8d89bc92a879f8a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a27c589ac548ca8d89bc92a879f8a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42763 } } }
I20260620 14:52:21.428977 29406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:21.429538 29407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75a27c589ac548ca8d89bc92a879f8a7. Latest consensus state: current_term: 1 leader_uuid: "75a27c589ac548ca8d89bc92a879f8a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a27c589ac548ca8d89bc92a879f8a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42763 } } }
I20260620 14:52:21.430106 29407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:21.429970 29423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:21.435233 29423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:21.439914 29403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:21.429486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40977 (local address 127.0.0.1:42763)
0620 14:52:21.429681 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:21.429686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:21.430053 (+ 367us) server_negotiation.cc:407] Connection header received
0620 14:52:21.430198 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:21.430201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:21.430263 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:21.430384 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:21.431337 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.432128 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.433306 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.433681 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.433842 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:21.435755 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:21.435760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:21.435763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:21.435791 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:21.437745 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:21.438392 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:21.438396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:21.438398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:21.438494 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:21.438889 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:21.438893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:21.438895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:21.439129 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:21.439300 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:21.439626 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 14:52:21.439736 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 14:52:21.442610 29423 catalog_manager.cc:1347] Generated new cluster ID: 482a20eb07c54d93917fe9eb2159ffae
I20260620 14:52:21.442689 29423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:21.473277 29423 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:21.474833 29423 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:21.482842 29423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75a27c589ac548ca8d89bc92a879f8a7: Generated new TSK 0
I20260620 14:52:21.483695 29423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:21.488360 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 29332
W20260620 14:52:21.493899 29317 connection.cc:535] client connection to 127.0.0.1:42763 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42763): Connection reset by peer (error 104)
2026-06-20T14:52:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1483 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:21 dist-test-slave-65tz krb5kdc[29431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:21 dist-test-slave-65tz krb5kdc[29431](info): set up 2 sockets
Jun 20 14:52:21 dist-test-slave-65tz krb5kdc[29431](info): commencing operation
krb5kdc: starting...
W20260620 14:52:22.534345 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:22 dist-test-slave-65tz krb5kdc[29431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:22.667233 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42489
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:22.799784 29444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:22.800110 29444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:22.800158 29444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:22.804414 29444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:22.804486 29444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:22.804507 29444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:22.804525 29444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:22.804540 29444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:22.810057 29444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42489
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:22.811362 29444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:22.812532 29444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:22.822677 29449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:22.822710 29452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:22.822770 29450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:22.823110 29444 server_base.cc:1034] running on GCE node
I20260620 14:52:22.823715 29444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:22.825011 29444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:22.826231 29444 hybrid_clock.cc:648] HybridClock initialized: now 1781967142826202 us; error 40 us; skew 500 ppm
Jun 20 14:52:22 dist-test-slave-65tz krb5kdc[29431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:22.829540 29444 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:22.830848 29444 webserver.cc:460] Webserver started at http://127.0.0.1:46235/ using document root <none> and password file <none>
I20260620 14:52:22.831593 29444 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:22.831693 29444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:22.831977 29444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:22.834555 29444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "3679928cadd74ef39d6e9ce7dc87726e"
format_stamp: "Formatted at 2026-06-20 14:52:22 on dist-test-slave-65tz"
server_key: "302db37ac05cf606cf09606c38b6baf1"
server_key_iv: "c7ea9db6db56447708c706e0a02e9a28"
server_key_version: "encryptionkey@0"
I20260620 14:52:22.835196 29444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "3679928cadd74ef39d6e9ce7dc87726e"
format_stamp: "Formatted at 2026-06-20 14:52:22 on dist-test-slave-65tz"
server_key: "302db37ac05cf606cf09606c38b6baf1"
server_key_iv: "c7ea9db6db56447708c706e0a02e9a28"
server_key_version: "encryptionkey@0"
I20260620 14:52:22.839833 29444 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:52:22.842861 29459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:22.844338 29444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:22.844499 29444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "3679928cadd74ef39d6e9ce7dc87726e"
format_stamp: "Formatted at 2026-06-20 14:52:22 on dist-test-slave-65tz"
server_key: "302db37ac05cf606cf09606c38b6baf1"
server_key_iv: "c7ea9db6db56447708c706e0a02e9a28"
server_key_version: "encryptionkey@0"
I20260620 14:52:22.844635 29444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:22.863739 29444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:22.867827 29444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:22.868072 29444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:22.877338 29444 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42489
I20260620 14:52:22.877360 29511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42489 every 8 connection(s)
I20260620 14:52:22.878654 29444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:22.881630 29512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:22.885138 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 29444
I20260620 14:52:22.885315 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:22.885622 9079 external_mini_cluster.cc:1428] Setting key 1a079950ea76dc2ce5234a46129c90db
I20260620 14:52:22.887979 29512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e: Bootstrap starting.
I20260620 14:52:22.890286 29512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:22.891052 29512 log.cc:824] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:22.893452 29512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e: No bootstrap required, opened a new log
I20260620 14:52:22.897019 29515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:22.887334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52521 (local address 127.0.0.1:42489)
0620 14:52:22.887859 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:22.887879 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:22.887926 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:52:22.888827 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:22.888848 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:22.889307 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:22.889719 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:22.890260 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:22.891564 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:22.892896 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:22.893462 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:22.893671 (+ 209us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:22.894703 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:22.894709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:22.894728 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:22.895298 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:22.895371 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:22.895401 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:22.895537 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:22.895836 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 14:52:22.896048 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":165,"threads_started":1}
I20260620 14:52:22.898022 29512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3679928cadd74ef39d6e9ce7dc87726e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42489 } }
I20260620 14:52:22.898269 29512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:22.898355 29512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3679928cadd74ef39d6e9ce7dc87726e, State: Initialized, Role: FOLLOWER
I20260620 14:52:22.898870 29512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3679928cadd74ef39d6e9ce7dc87726e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42489 } }
I20260620 14:52:22.899022 29512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:22.899091 29512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:22.899191 29512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:22.900295 29512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3679928cadd74ef39d6e9ce7dc87726e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42489 } }
I20260620 14:52:22.900650 29512 leader_election.cc:302] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3679928cadd74ef39d6e9ce7dc87726e; no voters:
I20260620 14:52:22.900780 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:22.901005 29512 leader_election.cc:288] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:22.901158 29517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:22.901511 29517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e [term 1 LEADER]: Becoming Leader. State: Replica: 3679928cadd74ef39d6e9ce7dc87726e, State: Running, Role: LEADER
I20260620 14:52:22.901971 29517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3679928cadd74ef39d6e9ce7dc87726e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42489 } }
I20260620 14:52:22.902483 29512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:22.904007 29519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3679928cadd74ef39d6e9ce7dc87726e. Latest consensus state: current_term: 1 leader_uuid: "3679928cadd74ef39d6e9ce7dc87726e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3679928cadd74ef39d6e9ce7dc87726e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42489 } } }
I20260620 14:52:22.904145 29519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:22.904611 29518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3679928cadd74ef39d6e9ce7dc87726e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3679928cadd74ef39d6e9ce7dc87726e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42489 } } }
I20260620 14:52:22.904702 29518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e [sys.catalog]: This master's current role is: LEADER
W20260620 14:52:22.907948 29537 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:52:22.908046 29537 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:52:22.908162 29533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:22.909025 29515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:22.903274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41601 (local address 127.0.0.1:42489)
0620 14:52:22.903426 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:22.903431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:22.903796 (+ 365us) server_negotiation.cc:407] Connection header received
0620 14:52:22.903995 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:22.903999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:22.904051 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:22.904119 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:22.904625 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:22.905435 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:22.906886 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:22.907217 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:22.907332 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:22.908490 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:22.908494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:22.908497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:22.908511 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:22.908542 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:22.908556 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:22.908663 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:22.908851 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 14:52:22.908890 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:52:22.909185 29533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:22.915551 29533 catalog_manager.cc:1347] Generated new cluster ID: 683c7682a9b741d38f92bdfff009549c
I20260620 14:52:22.915697 29533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:22.932566 29533 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:22.934057 29533 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:22.948314 29533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3679928cadd74ef39d6e9ce7dc87726e: Generated new TSK 0
I20260620 14:52:22.949465 29533 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:22.959342 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 29444
W20260620 14:52:22.965960 29429 connection.cc:535] client connection to 127.0.0.1:42489 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42489): Connection reset by peer (error 104)
2026-06-20T14:52:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1471 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:23 dist-test-slave-65tz krb5kdc[29543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:23 dist-test-slave-65tz krb5kdc[29543](info): set up 2 sockets
Jun 20 14:52:23 dist-test-slave-65tz krb5kdc[29543](info): commencing operation
krb5kdc: starting...
W20260620 14:52:24.014384 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:24 dist-test-slave-65tz krb5kdc[29543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:24.148388 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:24.264014 29556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:24.264271 29556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:24.264319 29556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:24.268353 29556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:24.268410 29556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:24.268430 29556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:24.268447 29556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:24.268463 29556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:24.273895 29556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42385
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:24.275022 29556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:24.276054 29556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:24.285174 29561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:24.285233 29564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:24.285347 29562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:24.286077 29556 server_base.cc:1034] running on GCE node
I20260620 14:52:24.286691 29556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:24.287868 29556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:24.289085 29556 hybrid_clock.cc:648] HybridClock initialized: now 1781967144289058 us; error 48 us; skew 500 ppm
Jun 20 14:52:24 dist-test-slave-65tz krb5kdc[29543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:24.295128 29556 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:24.296167 29556 webserver.cc:460] Webserver started at http://127.0.0.1:41681/ using document root <none> and password file <none>
I20260620 14:52:24.296779 29556 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:24.296872 29556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:24.297101 29556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:24.299259 29556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "c913859f3d6c4b1d8c8e5f2fb3a1bbf2"
format_stamp: "Formatted at 2026-06-20 14:52:24 on dist-test-slave-65tz"
server_key: "857d91a5464f5a12553784e9afd530d9"
server_key_iv: "4fcbd0a0505b71fad472451d2d780878"
server_key_version: "encryptionkey@0"
I20260620 14:52:24.299860 29556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "c913859f3d6c4b1d8c8e5f2fb3a1bbf2"
format_stamp: "Formatted at 2026-06-20 14:52:24 on dist-test-slave-65tz"
server_key: "857d91a5464f5a12553784e9afd530d9"
server_key_iv: "4fcbd0a0505b71fad472451d2d780878"
server_key_version: "encryptionkey@0"
I20260620 14:52:24.304008 29556 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 14:52:24.307123 29571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:24.308526 29556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:24.308681 29556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "c913859f3d6c4b1d8c8e5f2fb3a1bbf2"
format_stamp: "Formatted at 2026-06-20 14:52:24 on dist-test-slave-65tz"
server_key: "857d91a5464f5a12553784e9afd530d9"
server_key_iv: "4fcbd0a0505b71fad472451d2d780878"
server_key_version: "encryptionkey@0"
I20260620 14:52:24.308802 29556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:24.313532 29556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:24.316911 29556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:24.317106 29556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:24.324771 29556 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42385
I20260620 14:52:24.324796 29623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42385 every 8 connection(s)
I20260620 14:52:24.325776 29556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:24.329242 29624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:24.335179 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 29556
I20260620 14:52:24.335367 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:24.335672 9079 external_mini_cluster.cc:1428] Setting key af57bb8f6c6570387f1daec385ff1af3
I20260620 14:52:24.336018 29624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2: Bootstrap starting.
I20260620 14:52:24.338984 29624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:24.340034 29624 log.cc:824] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:24.341861 29627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:24.337182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53817 (local address 127.0.0.1:42385)
0620 14:52:24.337652 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:24.337667 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:24.337705 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:52:24.338572 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:24.338590 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:24.338904 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:24.339263 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:24.340249 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:24.340254 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:24.340265 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:24.340276 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:24.340334 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:24.340355 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:24.340489 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:24.340741 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 14:52:24.340961 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":144,"threads_started":1}
I20260620 14:52:24.343003 29624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2: No bootstrap required, opened a new log
I20260620 14:52:24.345320 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:24.348901 29624 raft_consensus.cc:348] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c913859f3d6c4b1d8c8e5f2fb3a1bbf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42385 } }
I20260620 14:52:24.349367 29624 raft_consensus.cc:374] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:24.349442 29624 raft_consensus.cc:729] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c913859f3d6c4b1d8c8e5f2fb3a1bbf2, State: Initialized, Role: FOLLOWER
I20260620 14:52:24.350131 29624 consensus_queue.cc:260] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c913859f3d6c4b1d8c8e5f2fb3a1bbf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42385 } }
I20260620 14:52:24.350353 29627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:24.348308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41657 (local address 127.0.0.1:42385)
0620 14:52:24.348449 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:24.348453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:24.348687 (+ 234us) server_negotiation.cc:407] Connection header received
0620 14:52:24.348855 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:24.348857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:24.348913 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:24.348987 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:24.349934 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:24.349936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:24.349938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:24.349949 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:24.349974 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:24.349983 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:24.350070 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:24.350194 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 14:52:24.350225 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:52:24.350358 29624 raft_consensus.cc:388] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:24.350561 29624 raft_consensus.cc:482] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:24.350692 29624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:24.351989 29624 raft_consensus.cc:504] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c913859f3d6c4b1d8c8e5f2fb3a1bbf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42385 } }
I20260620 14:52:24.352406 29624 leader_election.cc:302] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c913859f3d6c4b1d8c8e5f2fb3a1bbf2; no voters:
I20260620 14:52:24.352737 29624 leader_election.cc:288] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:24.352952 29634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:24.353286 29634 raft_consensus.cc:686] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2 [term 1 LEADER]: Becoming Leader. State: Replica: c913859f3d6c4b1d8c8e5f2fb3a1bbf2, State: Running, Role: LEADER
I20260620 14:52:24.353713 29634 consensus_queue.cc:237] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c913859f3d6c4b1d8c8e5f2fb3a1bbf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42385 } }
I20260620 14:52:24.354157 29624 sys_catalog.cc:564] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:24.356060 29636 sys_catalog.cc:455] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c913859f3d6c4b1d8c8e5f2fb3a1bbf2. Latest consensus state: current_term: 1 leader_uuid: "c913859f3d6c4b1d8c8e5f2fb3a1bbf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c913859f3d6c4b1d8c8e5f2fb3a1bbf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42385 } } }
I20260620 14:52:24.356215 29636 sys_catalog.cc:458] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:24.356714 29635 sys_catalog.cc:455] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c913859f3d6c4b1d8c8e5f2fb3a1bbf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c913859f3d6c4b1d8c8e5f2fb3a1bbf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42385 } } }
I20260620 14:52:24.356873 29635 sys_catalog.cc:458] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2 [sys.catalog]: This master's current role is: LEADER
W20260620 14:52:24.358907 29648 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:52:24.358979 29648 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:52:24.359063 29649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:24.359810 29649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:24.366305 29649 catalog_manager.cc:1347] Generated new cluster ID: 7488d6cf80784a30b88dc2075b06e8e6
I20260620 14:52:24.366384 29649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:24.374593 29649 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:24.375052 29649 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:24.383138 29649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c913859f3d6c4b1d8c8e5f2fb3a1bbf2: Generated new TSK 0
I20260620 14:52:24.384081 29649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:24.397145 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 29556
2026-06-20T14:52:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:24 dist-test-slave-65tz krb5kdc[29655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:24 dist-test-slave-65tz krb5kdc[29655](info): set up 2 sockets
Jun 20 14:52:24 dist-test-slave-65tz krb5kdc[29655](info): commencing operation
krb5kdc: starting...
W20260620 14:52:25.482372 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:25 dist-test-slave-65tz krb5kdc[29655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:25.614305 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34627
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:25.743651 29668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:25.743966 29668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:25.744014 29668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:25.748256 29668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:25.748334 29668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:25.748355 29668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:25.748373 29668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:25.748389 29668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:25.754820 29668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:25.756249 29668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:25.757515 29668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:25.767920 29673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:25.768079 29674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:25.768132 29668 server_base.cc:1034] running on GCE node
W20260620 14:52:25.768291 29676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:25.768947 29668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:25.770320 29668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:25.771523 29668 hybrid_clock.cc:648] HybridClock initialized: now 1781967145771509 us; error 35 us; skew 500 ppm
Jun 20 14:52:25 dist-test-slave-65tz krb5kdc[29655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:25.774428 29668 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:25.775415 29668 webserver.cc:460] Webserver started at http://127.0.0.1:40879/ using document root <none> and password file <none>
I20260620 14:52:25.775960 29668 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:25.776022 29668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:25.776211 29668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:25.778393 29668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "99e6869637a747c28171c3cd66e67271"
format_stamp: "Formatted at 2026-06-20 14:52:25 on dist-test-slave-65tz"
server_key: "9d732fb6f102387f2d9fbe4e5176ffd9"
server_key_iv: "8f6fc3c3ae58218357f008f2a0027817"
server_key_version: "encryptionkey@0"
I20260620 14:52:25.778901 29668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "99e6869637a747c28171c3cd66e67271"
format_stamp: "Formatted at 2026-06-20 14:52:25 on dist-test-slave-65tz"
server_key: "9d732fb6f102387f2d9fbe4e5176ffd9"
server_key_iv: "8f6fc3c3ae58218357f008f2a0027817"
server_key_version: "encryptionkey@0"
I20260620 14:52:25.783056 29668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:52:25.785727 29683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:25.787276 29668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:52:25.787391 29668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "99e6869637a747c28171c3cd66e67271"
format_stamp: "Formatted at 2026-06-20 14:52:25 on dist-test-slave-65tz"
server_key: "9d732fb6f102387f2d9fbe4e5176ffd9"
server_key_iv: "8f6fc3c3ae58218357f008f2a0027817"
server_key_version: "encryptionkey@0"
I20260620 14:52:25.787542 29668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:25.804023 29668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:25.806974 29668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:25.807183 29668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:25.814832 29668 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34627
I20260620 14:52:25.815071 29735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34627 every 8 connection(s)
I20260620 14:52:25.815830 29668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:25.819687 29736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:25.821516 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 29668
I20260620 14:52:25.821686 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:25.822096 9079 external_mini_cluster.cc:1428] Setting key b759059cdb28125507b594647b5cd5f3
I20260620 14:52:25.828158 29736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271: Bootstrap starting.
I20260620 14:52:25.831501 29736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:25.832520 29736 log.cc:824] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:52:25 dist-test-slave-65tz krb5kdc[29655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:52:25.835848 29736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271: No bootstrap required, opened a new log
I20260620 14:52:25.839565 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:25.823894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36197 (local address 127.0.0.1:34627)
0620 14:52:25.824420 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:25.824445 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:25.824499 (+ 54us) server_negotiation.cc:407] Connection header received
0620 14:52:25.825521 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:25.825551 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:25.826125 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:25.826581 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:25.827306 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.828960 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.829877 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.830653 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.830839 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:25.833462 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:25.833469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:25.833488 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:25.833521 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:25.835812 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:25.836320 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:25.836329 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:25.836339 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:25.836502 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:25.836785 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:25.836788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:25.836790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:25.837417 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:25.837622 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:25.837912 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 14:52:25.838293 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":116,"threads_started":1}
I20260620 14:52:25.841229 29736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99e6869637a747c28171c3cd66e67271" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34627 } }
I20260620 14:52:25.841514 29736 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:25.841578 29736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e6869637a747c28171c3cd66e67271, State: Initialized, Role: FOLLOWER
I20260620 14:52:25.842177 29736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99e6869637a747c28171c3cd66e67271" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34627 } }
I20260620 14:52:25.842362 29736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:25.842429 29736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:25.842531 29736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:25.843241 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:25.843859 29736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99e6869637a747c28171c3cd66e67271" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34627 } }
I20260620 14:52:25.844247 29736 leader_election.cc:302] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99e6869637a747c28171c3cd66e67271; no voters:
I20260620 14:52:25.844758 29736 leader_election.cc:288] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:25.844863 29745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:25.845065 29745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271 [term 1 LEADER]: Becoming Leader. State: Replica: 99e6869637a747c28171c3cd66e67271, State: Running, Role: LEADER
I20260620 14:52:25.845443 29745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99e6869637a747c28171c3cd66e67271" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34627 } }
I20260620 14:52:25.846309 29736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:25.847359 29746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99e6869637a747c28171c3cd66e67271. Latest consensus state: current_term: 1 leader_uuid: "99e6869637a747c28171c3cd66e67271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99e6869637a747c28171c3cd66e67271" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34627 } } }
I20260620 14:52:25.847592 29746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:25.847915 29746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99e6869637a747c28171c3cd66e67271" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99e6869637a747c28171c3cd66e67271" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34627 } } }
I20260620 14:52:25.848060 29746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:25.849180 29755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:25.853091 29755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:25.854102 29739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:25.845598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32951 (local address 127.0.0.1:34627)
0620 14:52:25.845767 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:25.845775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:25.845789 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:52:25.845862 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:25.845867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:25.845921 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:25.845994 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:25.846804 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.847598 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.848697 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.848994 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.849119 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:25.850790 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:25.850794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:25.850797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:25.850821 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:25.852322 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:25.852734 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:25.852738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:25.852741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:25.852820 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:25.853087 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:25.853091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:25.853093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:25.853350 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:25.853492 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:25.853760 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:52:25.853902 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:52:25.859565 29755 catalog_manager.cc:1347] Generated new cluster ID: ef2bcd7881e1402d9ffff1dbfb0288ce
I20260620 14:52:25.859655 29755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:25.871811 29755 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:25.872705 29755 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:25.877532 29755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99e6869637a747c28171c3cd66e67271: Generated new TSK 0
I20260620 14:52:25.878230 29755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:25.885509 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 29668
2026-06-20T14:52:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1488 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:25 dist-test-slave-65tz krb5kdc[29767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:25 dist-test-slave-65tz krb5kdc[29767](info): set up 2 sockets
Jun 20 14:52:25 dist-test-slave-65tz krb5kdc[29767](info): commencing operation
krb5kdc: starting...
W20260620 14:52:26.970774 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:27 dist-test-slave-65tz krb5kdc[29767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:27.106881 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44239
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:27.221742 29780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:27.222031 29780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:27.222074 29780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:27.226068 29780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:27.226133 29780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:27.226153 29780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:27.226169 29780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:27.226186 29780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:27.231534 29780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:27.232613 29780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:27.233639 29780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:27.242766 29785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:27.242842 29786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:27.242785 29780 server_base.cc:1034] running on GCE node
W20260620 14:52:27.242952 29788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:27.243821 29780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:27.245313 29780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:27.246564 29780 hybrid_clock.cc:648] HybridClock initialized: now 1781967147246551 us; error 42 us; skew 500 ppm
Jun 20 14:52:27 dist-test-slave-65tz krb5kdc[29767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:27.250097 29780 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:27.251179 29780 webserver.cc:460] Webserver started at http://127.0.0.1:45091/ using document root <none> and password file <none>
I20260620 14:52:27.251909 29780 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:27.252020 29780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:27.252290 29780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:27.254860 29780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "2bfb2015e07546eb821feb93c5d1a4fa"
format_stamp: "Formatted at 2026-06-20 14:52:27 on dist-test-slave-65tz"
server_key: "81a4200671389c8a290268cee8d2406d"
server_key_iv: "256067a63574e1d61eb2ff216b15ac61"
server_key_version: "encryptionkey@0"
I20260620 14:52:27.255529 29780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "2bfb2015e07546eb821feb93c5d1a4fa"
format_stamp: "Formatted at 2026-06-20 14:52:27 on dist-test-slave-65tz"
server_key: "81a4200671389c8a290268cee8d2406d"
server_key_iv: "256067a63574e1d61eb2ff216b15ac61"
server_key_version: "encryptionkey@0"
I20260620 14:52:27.259948 29780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 14:52:27.262902 29795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:27.264439 29780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:52:27.264560 29780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "2bfb2015e07546eb821feb93c5d1a4fa"
format_stamp: "Formatted at 2026-06-20 14:52:27 on dist-test-slave-65tz"
server_key: "81a4200671389c8a290268cee8d2406d"
server_key_iv: "256067a63574e1d61eb2ff216b15ac61"
server_key_version: "encryptionkey@0"
I20260620 14:52:27.264655 29780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:27.284454 29780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:27.288420 29780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:27.288607 29780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:27.297415 29780 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44239
I20260620 14:52:27.297462 29847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44239 every 8 connection(s)
I20260620 14:52:27.298599 29780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:27.301728 29848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:27.303987 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 29780
I20260620 14:52:27.304159 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:27.304484 9079 external_mini_cluster.cc:1428] Setting key ab8e0a2c5b12b6a0032842e4c2f86a47
I20260620 14:52:27.308454 29848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa: Bootstrap starting.
I20260620 14:52:27.310940 29848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:27.311748 29848 log.cc:824] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:27.314399 29848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa: No bootstrap required, opened a new log
I20260620 14:52:27.314893 29851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:27.306170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53325 (local address 127.0.0.1:44239)
0620 14:52:27.306708 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:27.306727 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:27.306769 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:52:27.307615 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:27.307635 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:27.308067 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:27.308465 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:27.308931 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:27.310070 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:27.311001 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:27.311569 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:27.311750 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:27.312891 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:27.312904 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:27.312918 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:27.313370 (+ 452us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:27.313441 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:27.313467 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:27.313600 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:27.313778 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 14:52:27.314063 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":162,"threads_started":1}
I20260620 14:52:27.318360 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:27.319370 29848 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bfb2015e07546eb821feb93c5d1a4fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44239 } }
I20260620 14:52:27.319774 29848 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:27.319866 29848 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bfb2015e07546eb821feb93c5d1a4fa, State: Initialized, Role: FOLLOWER
I20260620 14:52:27.320488 29848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bfb2015e07546eb821feb93c5d1a4fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44239 } }
I20260620 14:52:27.320691 29848 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:27.320827 29848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:27.320995 29848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:27.322305 29848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bfb2015e07546eb821feb93c5d1a4fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44239 } }
I20260620 14:52:27.322736 29848 leader_election.cc:302] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bfb2015e07546eb821feb93c5d1a4fa; no voters:
I20260620 14:52:27.323167 29848 leader_election.cc:288] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:27.323256 29858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:27.323485 29858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa [term 1 LEADER]: Becoming Leader. State: Replica: 2bfb2015e07546eb821feb93c5d1a4fa, State: Running, Role: LEADER
I20260620 14:52:27.323904 29858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bfb2015e07546eb821feb93c5d1a4fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44239 } }
I20260620 14:52:27.324765 29848 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:27.325096 29851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:27.320605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36009 (local address 127.0.0.1:44239)
0620 14:52:27.320839 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:27.320845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:27.321024 (+ 179us) server_negotiation.cc:407] Connection header received
0620 14:52:27.321167 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:27.321170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:27.321215 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:27.321316 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:27.321573 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:27.322205 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:27.323146 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:27.323519 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:27.323621 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:27.324541 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:27.324545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:27.324547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:27.324559 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:27.324589 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:27.324599 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:27.324678 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:27.324946 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:52:27.324997 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260620 14:52:27.326221 29859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bfb2015e07546eb821feb93c5d1a4fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bfb2015e07546eb821feb93c5d1a4fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44239 } } }
I20260620 14:52:27.326361 29859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:27.326789 29867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:27.326778 29860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bfb2015e07546eb821feb93c5d1a4fa. Latest consensus state: current_term: 1 leader_uuid: "2bfb2015e07546eb821feb93c5d1a4fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bfb2015e07546eb821feb93c5d1a4fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44239 } } }
I20260620 14:52:27.326972 29860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:27.330148 29867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:27.336233 29867 catalog_manager.cc:1347] Generated new cluster ID: f8dc808c752243e5b4e773b2de735d0c
I20260620 14:52:27.336328 29867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:27.352480 29867 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:27.353773 29867 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:27.364171 29867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bfb2015e07546eb821feb93c5d1a4fa: Generated new TSK 0
I20260620 14:52:27.365226 29867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:27.371081 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 29780
2026-06-20T14:52:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1485 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:27 dist-test-slave-65tz krb5kdc[29879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:27 dist-test-slave-65tz krb5kdc[29879](info): set up 2 sockets
Jun 20 14:52:27 dist-test-slave-65tz krb5kdc[29879](info): commencing operation
krb5kdc: starting...
W20260620 14:52:28.418969 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:28 dist-test-slave-65tz krb5kdc[29879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:28.553059 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34423
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:28.686486 29892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:28.686906 29892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:28.686993 29892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:28.691753 29892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:28.691880 29892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:28.691918 29892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:28.691960 29892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:28.692003 29892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:28.698467 29892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:28.699946 29892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:28.701296 29892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:28.711375 29897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:28.711366 29898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:28.711400 29900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:28.711776 29892 server_base.cc:1034] running on GCE node
I20260620 14:52:28.712466 29892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:28.713949 29892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:28.715190 29892 hybrid_clock.cc:648] HybridClock initialized: now 1781967148715176 us; error 35 us; skew 500 ppm
Jun 20 14:52:28 dist-test-slave-65tz krb5kdc[29879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:28.718667 29892 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:28.719913 29892 webserver.cc:460] Webserver started at http://127.0.0.1:35213/ using document root <none> and password file <none>
I20260620 14:52:28.720678 29892 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:28.720755 29892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:28.721033 29892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:28.723801 29892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "89c00b89f26941ffb16bcba99e463cee"
format_stamp: "Formatted at 2026-06-20 14:52:28 on dist-test-slave-65tz"
server_key: "045b7c6541e98c2b3d1708a90dbdc4cf"
server_key_iv: "96b84a3921c07ade2fd48838e445a136"
server_key_version: "encryptionkey@0"
I20260620 14:52:28.724500 29892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "89c00b89f26941ffb16bcba99e463cee"
format_stamp: "Formatted at 2026-06-20 14:52:28 on dist-test-slave-65tz"
server_key: "045b7c6541e98c2b3d1708a90dbdc4cf"
server_key_iv: "96b84a3921c07ade2fd48838e445a136"
server_key_version: "encryptionkey@0"
I20260620 14:52:28.729414 29892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260620 14:52:28.732580 29907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:28.734223 29892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:28.734370 29892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "89c00b89f26941ffb16bcba99e463cee"
format_stamp: "Formatted at 2026-06-20 14:52:28 on dist-test-slave-65tz"
server_key: "045b7c6541e98c2b3d1708a90dbdc4cf"
server_key_iv: "96b84a3921c07ade2fd48838e445a136"
server_key_version: "encryptionkey@0"
I20260620 14:52:28.734500 29892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:28.739800 29892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:28.743301 29892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:28.743518 29892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:28.752360 29892 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34423
I20260620 14:52:28.752519 29959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34423 every 8 connection(s)
I20260620 14:52:28.753537 29892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:28.756768 29960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:28.761258 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 29892
I20260620 14:52:28.761448 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:28.761772 9079 external_mini_cluster.cc:1428] Setting key 2e71564f6bc3a601173d22832797eee5
I20260620 14:52:28.762897 29960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee: Bootstrap starting.
I20260620 14:52:28.765385 29960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:28.766225 29960 log.cc:824] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:28.768966 29960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee: No bootstrap required, opened a new log
I20260620 14:52:28.769207 29963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:28.763761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49941 (local address 127.0.0.1:34423)
0620 14:52:28.764387 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:28.764408 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:28.764457 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:52:28.765388 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:28.765421 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:28.765974 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:28.766424 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:28.767291 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:28.767300 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:28.767326 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:28.767338 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:28.767402 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:28.767425 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:28.767541 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:28.767706 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 14:52:28.767934 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":204,"threads_started":1}
I20260620 14:52:28.772956 29960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c00b89f26941ffb16bcba99e463cee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34423 } }
I20260620 14:52:28.773176 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:28.773264 29960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:28.773334 29960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c00b89f26941ffb16bcba99e463cee, State: Initialized, Role: FOLLOWER
I20260620 14:52:28.773761 29960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c00b89f26941ffb16bcba99e463cee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34423 } }
I20260620 14:52:28.773947 29960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:28.774037 29960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:28.774139 29960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:28.775244 29960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c00b89f26941ffb16bcba99e463cee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34423 } }
I20260620 14:52:28.775592 29960 leader_election.cc:302] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89c00b89f26941ffb16bcba99e463cee; no voters:
I20260620 14:52:28.775959 29960 leader_election.cc:288] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:28.776117 29969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:28.776409 29969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee [term 1 LEADER]: Becoming Leader. State: Replica: 89c00b89f26941ffb16bcba99e463cee, State: Running, Role: LEADER
I20260620 14:52:28.776907 29969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c00b89f26941ffb16bcba99e463cee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34423 } }
I20260620 14:52:28.777387 29960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:28.778483 29972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89c00b89f26941ffb16bcba99e463cee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c00b89f26941ffb16bcba99e463cee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34423 } } }
I20260620 14:52:28.778800 29972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:28.778491 29970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89c00b89f26941ffb16bcba99e463cee. Latest consensus state: current_term: 1 leader_uuid: "89c00b89f26941ffb16bcba99e463cee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c00b89f26941ffb16bcba99e463cee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34423 } } }
I20260620 14:52:28.779079 29970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:28.780616 29963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:28.776208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58325 (local address 127.0.0.1:34423)
0620 14:52:28.776390 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:28.776397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:28.776656 (+ 259us) server_negotiation.cc:407] Connection header received
0620 14:52:28.776801 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:28.776804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:28.776858 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:28.776955 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:28.780021 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:28.780026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:28.780029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:28.780044 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:28.780085 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:28.780100 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:28.780197 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:28.780376 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 14:52:28.780435 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 14:52:28.785449 29985 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:52:28.785557 29985 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:52:28.785640 29979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:28.786899 29979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:28.795147 29979 catalog_manager.cc:1347] Generated new cluster ID: 8abc79a714ed4db7ad1cc36edcff0639
I20260620 14:52:28.795239 29979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:28.855806 29979 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:28.856244 29979 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:28.863970 29979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89c00b89f26941ffb16bcba99e463cee: Generated new TSK 0
I20260620 14:52:28.864852 29979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:28.881145 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 29892
2026-06-20T14:52:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1512 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:28 dist-test-slave-65tz krb5kdc[29991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:28 dist-test-slave-65tz krb5kdc[29991](info): set up 2 sockets
Jun 20 14:52:28 dist-test-slave-65tz krb5kdc[29991](info): commencing operation
krb5kdc: starting...
W20260620 14:52:30.043069 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:30 dist-test-slave-65tz krb5kdc[29991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:30.181435 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40167
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:30.312124 30004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:30.312625 30004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:30.312692 30004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:30.316969 30004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:30.317045 30004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:30.317065 30004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:30.317085 30004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:30.317101 30004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:30.323035 30004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:30.324415 30004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:30.325676 30004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:30.336143 30009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:30.336259 30012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:30.336181 30010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:30.336171 30004 server_base.cc:1034] running on GCE node
I20260620 14:52:30.337121 30004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:30.338572 30004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:30.339802 30004 hybrid_clock.cc:648] HybridClock initialized: now 1781967150339784 us; error 63 us; skew 500 ppm
Jun 20 14:52:30 dist-test-slave-65tz krb5kdc[29991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:30.343016 30004 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:30.344106 30004 webserver.cc:460] Webserver started at http://127.0.0.1:44091/ using document root <none> and password file <none>
I20260620 14:52:30.344728 30004 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:30.344797 30004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:30.344985 30004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:30.347401 30004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "0c481b7168ca4baf8658da28ae2d9a62"
format_stamp: "Formatted at 2026-06-20 14:52:30 on dist-test-slave-65tz"
server_key: "9b3ead0850e5d9ddef7e74f53fb762c4"
server_key_iv: "f180b3f6ba71b8d47b14fa9083c9d716"
server_key_version: "encryptionkey@0"
I20260620 14:52:30.347993 30004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "0c481b7168ca4baf8658da28ae2d9a62"
format_stamp: "Formatted at 2026-06-20 14:52:30 on dist-test-slave-65tz"
server_key: "9b3ead0850e5d9ddef7e74f53fb762c4"
server_key_iv: "f180b3f6ba71b8d47b14fa9083c9d716"
server_key_version: "encryptionkey@0"
I20260620 14:52:30.352378 30004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:52:30.355324 30019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:30.356889 30004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:52:30.357045 30004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "0c481b7168ca4baf8658da28ae2d9a62"
format_stamp: "Formatted at 2026-06-20 14:52:30 on dist-test-slave-65tz"
server_key: "9b3ead0850e5d9ddef7e74f53fb762c4"
server_key_iv: "f180b3f6ba71b8d47b14fa9083c9d716"
server_key_version: "encryptionkey@0"
I20260620 14:52:30.357167 30004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:30.387722 30004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:30.391616 30004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:30.391897 30004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:30.400790 30004 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40167
I20260620 14:52:30.400810 30071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40167 every 8 connection(s)
I20260620 14:52:30.402050 30004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:30.405382 30072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:30.409747 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 30004
I20260620 14:52:30.409981 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:30.410368 9079 external_mini_cluster.cc:1428] Setting key b11487227acff3f7c5545edf159d48ee
I20260620 14:52:30.411897 30072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62: Bootstrap starting.
I20260620 14:52:30.414484 30072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:30.415362 30072 log.cc:824] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:30.418104 30072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62: No bootstrap required, opened a new log
Jun 20 14:52:30 dist-test-slave-65tz krb5kdc[29991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:52:30.423144 30072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c481b7168ca4baf8658da28ae2d9a62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40167 } }
I20260620 14:52:30.423436 30072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:30.423523 30072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c481b7168ca4baf8658da28ae2d9a62, State: Initialized, Role: FOLLOWER
I20260620 14:52:30.424103 30072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c481b7168ca4baf8658da28ae2d9a62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40167 } }
I20260620 14:52:30.424295 30072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:30.424389 30072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:30.424543 30072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:30.425642 30072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c481b7168ca4baf8658da28ae2d9a62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40167 } }
I20260620 14:52:30.426043 30072 leader_election.cc:302] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c481b7168ca4baf8658da28ae2d9a62; no voters:
I20260620 14:52:30.426502 30072 leader_election.cc:288] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:30.426694 30077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:30.427103 30077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62 [term 1 LEADER]: Becoming Leader. State: Replica: 0c481b7168ca4baf8658da28ae2d9a62, State: Running, Role: LEADER
I20260620 14:52:30.427551 30077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c481b7168ca4baf8658da28ae2d9a62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40167 } }
I20260620 14:52:30.428014 30072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:30.429615 30078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c481b7168ca4baf8658da28ae2d9a62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c481b7168ca4baf8658da28ae2d9a62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40167 } } }
I20260620 14:52:30.429761 30078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:30.430049 30078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c481b7168ca4baf8658da28ae2d9a62. Latest consensus state: current_term: 1 leader_uuid: "0c481b7168ca4baf8658da28ae2d9a62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c481b7168ca4baf8658da28ae2d9a62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40167 } } }
I20260620 14:52:30.430121 30078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:30.430183 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:30.412270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:57453 (local address 127.0.0.1:40167)
0620 14:52:30.413017 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:30.413042 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:30.413098 (+ 56us) server_negotiation.cc:407] Connection header received
0620 14:52:30.414178 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:30.414205 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:30.414742 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:30.415132 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:30.416106 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:30.417422 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:30.418821 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:30.419388 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:30.419536 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:30.423023 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:30.423029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:30.423047 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:30.423078 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:30.425333 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:30.426023 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:30.426030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:30.426037 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:30.426170 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:30.426768 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:30.426774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:30.426777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:30.427374 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:30.427642 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:30.428597 (+ 955us) server_negotiation.cc:299] Negotiation successful
0620 14:52:30.428875 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":152,"threads_started":1}
I20260620 14:52:30.430258 30089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:30.434134 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:30.434377 30089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:30.440819 30089 catalog_manager.cc:1347] Generated new cluster ID: c413262a9fd248498f0871d3b5dda072
I20260620 14:52:30.440887 30089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:30.449620 30075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:30.436447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:59715 (local address 127.0.0.1:40167)
0620 14:52:30.436591 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:30.436597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:30.438954 (+ 2357us) server_negotiation.cc:407] Connection header received
0620 14:52:30.439005 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:30.439009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:30.439068 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:30.439163 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:30.440058 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:30.441524 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:30.442830 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:30.443246 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:30.443380 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:30.445308 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:30.445313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:30.445316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:30.445348 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:30.447454 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:30.448112 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:30.448116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:30.448118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:30.448215 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:30.448584 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:30.448587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:30.448589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:30.448826 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:30.449006 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:30.449304 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 14:52:30.449422 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:52:30.456993 30089 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:30.458503 30089 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:30.466228 30089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c481b7168ca4baf8658da28ae2d9a62: Generated new TSK 0
I20260620 14:52:30.467265 30089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:30.472692 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 30004
W20260620 14:52:30.479646 29989 connection.cc:535] client connection to 127.0.0.1:40167 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40167): Connection reset by peer (error 104)
2026-06-20T14:52:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1590 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:30 dist-test-slave-65tz krb5kdc[30103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:30 dist-test-slave-65tz krb5kdc[30103](info): set up 2 sockets
Jun 20 14:52:30 dist-test-slave-65tz krb5kdc[30103](info): commencing operation
krb5kdc: starting...
W20260620 14:52:31.546465 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:31 dist-test-slave-65tz krb5kdc[30103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:31.676733 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43899
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:31.795423 30116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:31.795727 30116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:31.795770 30116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:31.799954 30116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:31.800027 30116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:31.800047 30116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:31.800065 30116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:31.800081 30116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:31.806351 30116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:31.807513 30116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:31.808617 30116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:31.818013 30124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:31.818150 30121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:31.818013 30122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:31.818490 30116 server_base.cc:1034] running on GCE node
I20260620 14:52:31.819213 30116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:31.820430 30116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:31.821667 30116 hybrid_clock.cc:648] HybridClock initialized: now 1781967151821631 us; error 54 us; skew 500 ppm
Jun 20 14:52:31 dist-test-slave-65tz krb5kdc[30103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:31.824796 30116 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:31.825851 30116 webserver.cc:460] Webserver started at http://127.0.0.1:40271/ using document root <none> and password file <none>
I20260620 14:52:31.826462 30116 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:31.826555 30116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:31.826795 30116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:31.828871 30116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "125db234af604b03939f57ff41964b1a"
format_stamp: "Formatted at 2026-06-20 14:52:31 on dist-test-slave-65tz"
server_key: "91cde07cc823bd0badd40ee29c3e2dcc"
server_key_iv: "d3b62c8b11f2b93da74b4ac4ef8dccc9"
server_key_version: "encryptionkey@0"
I20260620 14:52:31.829447 30116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "125db234af604b03939f57ff41964b1a"
format_stamp: "Formatted at 2026-06-20 14:52:31 on dist-test-slave-65tz"
server_key: "91cde07cc823bd0badd40ee29c3e2dcc"
server_key_iv: "d3b62c8b11f2b93da74b4ac4ef8dccc9"
server_key_version: "encryptionkey@0"
I20260620 14:52:31.833520 30116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:52:31.836320 30131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:31.837869 30116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:31.838019 30116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "125db234af604b03939f57ff41964b1a"
format_stamp: "Formatted at 2026-06-20 14:52:31 on dist-test-slave-65tz"
server_key: "91cde07cc823bd0badd40ee29c3e2dcc"
server_key_iv: "d3b62c8b11f2b93da74b4ac4ef8dccc9"
server_key_version: "encryptionkey@0"
I20260620 14:52:31.838140 30116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:31.851454 30116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:31.854851 30116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:31.855054 30116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:31.862737 30116 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43899
I20260620 14:52:31.862792 30183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43899 every 8 connection(s)
I20260620 14:52:31.863801 30116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:31.864495 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 30116
I20260620 14:52:31.864621 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:31.864969 9079 external_mini_cluster.cc:1428] Setting key bbe7ca56e209972187fe24c8b61407e6
I20260620 14:52:31.868216 30184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:31.875674 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:31.867032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:47625 (local address 127.0.0.1:43899)
0620 14:52:31.867563 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:31.867583 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:31.867624 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:52:31.868299 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:31.868316 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:31.868760 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:31.869071 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:31.869570 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:31.870659 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:31.872015 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:31.872508 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:31.872641 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:31.873768 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:31.873773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:31.873787 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:31.874281 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:31.874347 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:31.874372 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:31.874483 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:31.874707 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 14:52:31.874888 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":170,"threads_started":1}
I20260620 14:52:31.876441 30184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a: Bootstrap starting.
I20260620 14:52:31.879349 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:31.879791 30184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:31.880743 30184 log.cc:824] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:31.883055 30184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a: No bootstrap required, opened a new log
I20260620 14:52:31.887061 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:31.881565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:41019 (local address 127.0.0.1:43899)
0620 14:52:31.881695 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:31.881700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:31.883424 (+ 1724us) server_negotiation.cc:407] Connection header received
0620 14:52:31.883580 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:31.883583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:31.883638 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:31.883724 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:31.883980 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:31.884525 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:31.885325 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:31.885567 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:31.885676 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:31.886583 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:31.886587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:31.886590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:31.886602 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:31.886634 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:31.886644 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:31.886743 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:31.886867 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 14:52:31.886904 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:52:31.887226 30184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125db234af604b03939f57ff41964b1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43899 } }
I20260620 14:52:31.887609 30184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:31.887701 30184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125db234af604b03939f57ff41964b1a, State: Initialized, Role: FOLLOWER
I20260620 14:52:31.888252 30184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125db234af604b03939f57ff41964b1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43899 } }
I20260620 14:52:31.888479 30184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:31.888561 30184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:31.888693 30184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:31.890013 30184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125db234af604b03939f57ff41964b1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43899 } }
I20260620 14:52:31.890468 30184 leader_election.cc:302] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 125db234af604b03939f57ff41964b1a; no voters:
I20260620 14:52:31.890877 30184 leader_election.cc:288] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:31.891036 30194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:31.891341 30194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a [term 1 LEADER]: Becoming Leader. State: Replica: 125db234af604b03939f57ff41964b1a, State: Running, Role: LEADER
I20260620 14:52:31.891825 30194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125db234af604b03939f57ff41964b1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43899 } }
I20260620 14:52:31.892616 30184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:31.893924 30195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "125db234af604b03939f57ff41964b1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125db234af604b03939f57ff41964b1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43899 } } }
I20260620 14:52:31.893910 30196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 125db234af604b03939f57ff41964b1a. Latest consensus state: current_term: 1 leader_uuid: "125db234af604b03939f57ff41964b1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125db234af604b03939f57ff41964b1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43899 } } }
I20260620 14:52:31.894073 30195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:31.894073 30196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:31.894515 30203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:31.898016 30203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:31.904631 30203 catalog_manager.cc:1347] Generated new cluster ID: b019c1d0d1f944b4971342a204f20858
I20260620 14:52:31.904700 30203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:31.920714 30203 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:31.922511 30203 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:31.927289 30203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 125db234af604b03939f57ff41964b1a: Generated new TSK 0
I20260620 14:52:31.928303 30203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:31.938599 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 30116
W20260620 14:52:31.944442 30101 connection.cc:535] client connection to 127.0.0.1:43899 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43899): Connection reset by peer (error 104)
2026-06-20T14:52:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1465 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:32 dist-test-slave-65tz krb5kdc[30215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:32 dist-test-slave-65tz krb5kdc[30215](info): set up 2 sockets
Jun 20 14:52:32 dist-test-slave-65tz krb5kdc[30215](info): commencing operation
krb5kdc: starting...
W20260620 14:52:32.990901 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:33 dist-test-slave-65tz krb5kdc[30215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:33.127571 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37747
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:33.241086 30228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:33.241384 30228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:33.241461 30228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:33.245524 30228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:33.245615 30228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:33.245653 30228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:33.245710 30228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:33.245748 30228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:33.250883 30228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:33.251911 30228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:33.252892 30228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:33.261926 30233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:33.261937 30234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:33.261942 30236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:33.262364 30228 server_base.cc:1034] running on GCE node
I20260620 14:52:33.263151 30228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:33.264608 30228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:33.265908 30228 hybrid_clock.cc:648] HybridClock initialized: now 1781967153265865 us; error 59 us; skew 500 ppm
Jun 20 14:52:33 dist-test-slave-65tz krb5kdc[30215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:33.269157 30228 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:33.270305 30228 webserver.cc:460] Webserver started at http://127.0.0.1:41431/ using document root <none> and password file <none>
I20260620 14:52:33.270967 30228 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:33.271041 30228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:33.271292 30228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:33.273677 30228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "8d6ea06f165a4949a538080fcd639f0a"
format_stamp: "Formatted at 2026-06-20 14:52:33 on dist-test-slave-65tz"
server_key: "0897e4974120ba4398d2fb9ed703481a"
server_key_iv: "4a1accda3a7814a62888b6fa8f03eeba"
server_key_version: "encryptionkey@0"
I20260620 14:52:33.274367 30228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "8d6ea06f165a4949a538080fcd639f0a"
format_stamp: "Formatted at 2026-06-20 14:52:33 on dist-test-slave-65tz"
server_key: "0897e4974120ba4398d2fb9ed703481a"
server_key_iv: "4a1accda3a7814a62888b6fa8f03eeba"
server_key_version: "encryptionkey@0"
I20260620 14:52:33.278621 30228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:52:33.281541 30243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:33.283169 30228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:52:33.283344 30228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "8d6ea06f165a4949a538080fcd639f0a"
format_stamp: "Formatted at 2026-06-20 14:52:33 on dist-test-slave-65tz"
server_key: "0897e4974120ba4398d2fb9ed703481a"
server_key_iv: "4a1accda3a7814a62888b6fa8f03eeba"
server_key_version: "encryptionkey@0"
I20260620 14:52:33.283473 30228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:33.288938 30228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:33.292565 30228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:33.292826 30228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:33.301816 30228 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37747
I20260620 14:52:33.301859 30295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37747 every 8 connection(s)
I20260620 14:52:33.303018 30228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:33.305580 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 30228
I20260620 14:52:33.305754 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:33.306121 9079 external_mini_cluster.cc:1428] Setting key 22bdcebd6b0a9069b2f8d1b4fd296230
I20260620 14:52:33.306207 30296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:33.312672 30299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:33.307782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:43621 (local address 127.0.0.1:37747)
0620 14:52:33.308275 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:33.308293 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:33.308336 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:52:33.309483 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:33.309504 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:33.309944 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:33.310296 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:33.311250 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:33.311255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:33.311268 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:33.311278 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:33.311340 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:33.311360 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:33.311481 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:33.311614 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 14:52:33.311845 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":169,"threads_started":1}
I20260620 14:52:33.313450 30296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a: Bootstrap starting.
I20260620 14:52:33.315840 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:33.315912 30296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:33.316639 30296 log.cc:824] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:33.319031 30296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a: No bootstrap required, opened a new log
I20260620 14:52:33.320384 30299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:33.318460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:48379 (local address 127.0.0.1:37747)
0620 14:52:33.318583 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:33.318586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:33.318841 (+ 255us) server_negotiation.cc:407] Connection header received
0620 14:52:33.318999 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:33.319004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:33.319048 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:33.319147 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:33.319909 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:33.319912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:33.319914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:33.319924 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:33.319951 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:33.319960 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:33.320035 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:33.320164 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 14:52:33.320206 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:52:33.323158 30296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6ea06f165a4949a538080fcd639f0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37747 } }
I20260620 14:52:33.323371 30296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:33.323458 30296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6ea06f165a4949a538080fcd639f0a, State: Initialized, Role: FOLLOWER
I20260620 14:52:33.323889 30296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6ea06f165a4949a538080fcd639f0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37747 } }
I20260620 14:52:33.324034 30296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:33.324107 30296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:33.324226 30296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:33.325131 30296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6ea06f165a4949a538080fcd639f0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37747 } }
I20260620 14:52:33.325546 30296 leader_election.cc:302] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d6ea06f165a4949a538080fcd639f0a; no voters:
I20260620 14:52:33.325946 30296 leader_election.cc:288] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:33.326146 30306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:33.326515 30306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a [term 1 LEADER]: Becoming Leader. State: Replica: 8d6ea06f165a4949a538080fcd639f0a, State: Running, Role: LEADER
I20260620 14:52:33.326858 30306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6ea06f165a4949a538080fcd639f0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37747 } }
I20260620 14:52:33.327215 30296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:33.328768 30308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d6ea06f165a4949a538080fcd639f0a. Latest consensus state: current_term: 1 leader_uuid: "8d6ea06f165a4949a538080fcd639f0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6ea06f165a4949a538080fcd639f0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37747 } } }
I20260620 14:52:33.328832 30307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d6ea06f165a4949a538080fcd639f0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d6ea06f165a4949a538080fcd639f0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37747 } } }
I20260620 14:52:33.329066 30308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:33.329097 30307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:33.329550 30320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:33.332456 30320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:33.338531 30320 catalog_manager.cc:1347] Generated new cluster ID: d29b51dcca6d462cbf25e29365ceeee7
I20260620 14:52:33.338608 30320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:33.353004 30320 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:33.353466 30320 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:33.359977 30320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d6ea06f165a4949a538080fcd639f0a: Generated new TSK 0
I20260620 14:52:33.360776 30320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:33.363807 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 30228
2026-06-20T14:52:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:33 dist-test-slave-65tz krb5kdc[30327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:33 dist-test-slave-65tz krb5kdc[30327](info): set up 2 sockets
Jun 20 14:52:33 dist-test-slave-65tz krb5kdc[30327](info): commencing operation
krb5kdc: starting...
W20260620 14:52:34.482467 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:34 dist-test-slave-65tz krb5kdc[30327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:34.621465 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:34.738317 30340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:34.738672 30340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:34.738752 30340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:34.743332 30340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:34.743489 30340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:34.743538 30340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:34.743623 30340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:34.743667 30340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:34.749454 30340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46273
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:34.750723 30340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:34.751796 30340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:34.761361 30346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:34.761417 30345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:34.761487 30340 server_base.cc:1034] running on GCE node
W20260620 14:52:34.761396 30348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:34.762408 30340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:34.763800 30340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:34.765013 30340 hybrid_clock.cc:648] HybridClock initialized: now 1781967154764990 us; error 43 us; skew 500 ppm
Jun 20 14:52:34 dist-test-slave-65tz krb5kdc[30327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:34.772259 30340 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:34.773612 30340 webserver.cc:460] Webserver started at http://127.0.0.1:36287/ using document root <none> and password file <none>
I20260620 14:52:34.774425 30340 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:34.774501 30340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:34.774777 30340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:34.777563 30340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "2c52e4ab816c45c49e9753d252642773"
format_stamp: "Formatted at 2026-06-20 14:52:34 on dist-test-slave-65tz"
server_key: "8e12cede8c58a7bd3f28ddac92e13f4b"
server_key_iv: "39614ee754cc4eb9f20053f33f3fec02"
server_key_version: "encryptionkey@0"
I20260620 14:52:34.778504 30340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "2c52e4ab816c45c49e9753d252642773"
format_stamp: "Formatted at 2026-06-20 14:52:34 on dist-test-slave-65tz"
server_key: "8e12cede8c58a7bd3f28ddac92e13f4b"
server_key_iv: "39614ee754cc4eb9f20053f33f3fec02"
server_key_version: "encryptionkey@0"
I20260620 14:52:34.785317 30340 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 14:52:34.788942 30355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:34.790861 30340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:34.791066 30340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "2c52e4ab816c45c49e9753d252642773"
format_stamp: "Formatted at 2026-06-20 14:52:34 on dist-test-slave-65tz"
server_key: "8e12cede8c58a7bd3f28ddac92e13f4b"
server_key_iv: "39614ee754cc4eb9f20053f33f3fec02"
server_key_version: "encryptionkey@0"
I20260620 14:52:34.791246 30340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:34.809129 30340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:34.813359 30340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:34.813647 30340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:34.822832 30340 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46273
I20260620 14:52:34.822862 30407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46273 every 8 connection(s)
I20260620 14:52:34.824069 30340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:34.827330 30408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:34.830210 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 30340
I20260620 14:52:34.830369 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:34.830703 9079 external_mini_cluster.cc:1428] Setting key a438e4f4a6728d971502f786b8cb1561
I20260620 14:52:34.834676 30408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773: Bootstrap starting.
I20260620 14:52:34.837384 30408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:34.838260 30408 log.cc:824] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:34.840373 30408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773: No bootstrap required, opened a new log
Jun 20 14:52:34 dist-test-slave-65tz krb5kdc[30327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:52:34.844110 30408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c52e4ab816c45c49e9753d252642773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46273 } }
I20260620 14:52:34.844445 30408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:34.844527 30408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c52e4ab816c45c49e9753d252642773, State: Initialized, Role: FOLLOWER
I20260620 14:52:34.844991 30408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c52e4ab816c45c49e9753d252642773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46273 } }
I20260620 14:52:34.845119 30408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:34.845212 30408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:34.845361 30408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:34.846662 30408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c52e4ab816c45c49e9753d252642773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46273 } }
I20260620 14:52:34.847034 30408 leader_election.cc:302] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c52e4ab816c45c49e9753d252642773; no voters:
I20260620 14:52:34.847429 30408 leader_election.cc:288] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:34.847512 30413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:34.847790 30413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773 [term 1 LEADER]: Becoming Leader. State: Replica: 2c52e4ab816c45c49e9753d252642773, State: Running, Role: LEADER
I20260620 14:52:34.848146 30413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c52e4ab816c45c49e9753d252642773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46273 } }
I20260620 14:52:34.848803 30408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:34.849931 30414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c52e4ab816c45c49e9753d252642773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c52e4ab816c45c49e9753d252642773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46273 } } }
I20260620 14:52:34.850080 30414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:34.850394 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:34.832540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:43805 (local address 127.0.0.1:46273)
0620 14:52:34.833045 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:34.833066 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:34.833118 (+ 52us) server_negotiation.cc:407] Connection header received
0620 14:52:34.834145 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:34.834169 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:34.834655 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:34.835077 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:34.836025 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:34.837285 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:34.838492 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:34.839073 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:34.839322 (+ 249us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:34.842678 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:34.842684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:34.842703 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:34.842733 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:34.845654 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:34.846348 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:34.846354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:34.846360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:34.846488 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:34.846931 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:34.846934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:34.846935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:34.847473 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:34.848609 (+ 1136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:34.848817 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 14:52:34.849108 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":136,"threads_started":1}
I20260620 14:52:34.850415 30415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c52e4ab816c45c49e9753d252642773. Latest consensus state: current_term: 1 leader_uuid: "2c52e4ab816c45c49e9753d252642773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c52e4ab816c45c49e9753d252642773" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46273 } } }
I20260620 14:52:34.850875 30415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:34.854574 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 14:52:34.855086 30428 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:52:34.855191 30428 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:52:34.855353 30422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:34.856105 30422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:34.863124 30422 catalog_manager.cc:1347] Generated new cluster ID: 1a117f6da206439b95cdf22a0dd667b8
I20260620 14:52:34.863278 30422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:34.868119 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:34.858610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:51355 (local address 127.0.0.1:46273)
0620 14:52:34.858759 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:34.858764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:34.858904 (+ 140us) server_negotiation.cc:407] Connection header received
0620 14:52:34.859077 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:34.859081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:34.859133 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:34.859217 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:34.859881 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:34.860443 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:34.861466 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:34.861829 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:34.861943 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:34.864061 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:34.864064 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:34.864067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:34.864093 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:34.866045 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:34.866693 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:34.866696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:34.866698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:34.866788 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:34.867161 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:34.867164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:34.867166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:34.867361 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:34.867500 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:34.867842 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 14:52:34.867956 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:52:34.881979 30422 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:34.883639 30422 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:34.891749 30422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c52e4ab816c45c49e9753d252642773: Generated new TSK 0
I20260620 14:52:34.892944 30422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:34.903218 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 30340
W20260620 14:52:34.909732 30325 connection.cc:535] client connection to 127.0.0.1:46273 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46273): Connection reset by peer (error 104)
2026-06-20T14:52:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1539 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:34 dist-test-slave-65tz krb5kdc[30439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:34 dist-test-slave-65tz krb5kdc[30439](info): set up 2 sockets
Jun 20 14:52:34 dist-test-slave-65tz krb5kdc[30439](info): commencing operation
krb5kdc: starting...
W20260620 14:52:35.958443 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:36 dist-test-slave-65tz krb5kdc[30439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:36.098433 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44535
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:36.210155 30452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:36.210776 30452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:36.210906 30452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:36.215051 30452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:36.215149 30452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:36.215188 30452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:36.215226 30452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:36.215260 30452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:36.220624 30452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44535
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:36.221815 30452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:36.222903 30452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:36.232218 30460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:36.232251 30458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:36.232213 30457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:36.232647 30452 server_base.cc:1034] running on GCE node
I20260620 14:52:36.233436 30452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:36.234972 30452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:36.236161 30452 hybrid_clock.cc:648] HybridClock initialized: now 1781967156236127 us; error 55 us; skew 500 ppm
Jun 20 14:52:36 dist-test-slave-65tz krb5kdc[30439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:36.239419 30452 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:36.240485 30452 webserver.cc:460] Webserver started at http://127.0.0.1:36527/ using document root <none> and password file <none>
I20260620 14:52:36.241132 30452 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:36.241194 30452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:36.241391 30452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:36.243875 30452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "14d2929f66d0489995e516e8d53e6561"
format_stamp: "Formatted at 2026-06-20 14:52:36 on dist-test-slave-65tz"
server_key: "086bec478e6d484ecf395097e59bac54"
server_key_iv: "a180a5d99e019d218fa14102bd8c9a0a"
server_key_version: "encryptionkey@0"
I20260620 14:52:36.244473 30452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "14d2929f66d0489995e516e8d53e6561"
format_stamp: "Formatted at 2026-06-20 14:52:36 on dist-test-slave-65tz"
server_key: "086bec478e6d484ecf395097e59bac54"
server_key_iv: "a180a5d99e019d218fa14102bd8c9a0a"
server_key_version: "encryptionkey@0"
I20260620 14:52:36.248988 30452 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 14:52:36.252046 30467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:36.253626 30452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:52:36.253898 30452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "14d2929f66d0489995e516e8d53e6561"
format_stamp: "Formatted at 2026-06-20 14:52:36 on dist-test-slave-65tz"
server_key: "086bec478e6d484ecf395097e59bac54"
server_key_iv: "a180a5d99e019d218fa14102bd8c9a0a"
server_key_version: "encryptionkey@0"
I20260620 14:52:36.254035 30452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:36.282235 30452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:36.286631 30452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:36.286906 30452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:36.296178 30519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44535 every 8 connection(s)
I20260620 14:52:36.296181 30452 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44535
I20260620 14:52:36.297740 30452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:36.300946 30520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:36.306770 30520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561: Bootstrap starting.
I20260620 14:52:36.306841 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 30452
I20260620 14:52:36.306972 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:36.307271 9079 external_mini_cluster.cc:1428] Setting key 2241c66da4476264e5137abdcfb1867e
I20260620 14:52:36.309533 30520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:36.310366 30520 log.cc:824] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:36.312465 30520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561: No bootstrap required, opened a new log
I20260620 14:52:36.316891 30520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d2929f66d0489995e516e8d53e6561" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44535 } }
I20260620 14:52:36.317164 30520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:36.317247 30520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d2929f66d0489995e516e8d53e6561, State: Initialized, Role: FOLLOWER
I20260620 14:52:36.317729 30520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d2929f66d0489995e516e8d53e6561" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44535 } }
I20260620 14:52:36.317957 30520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:36.318053 30520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:36.318159 30520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:36.319110 30520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d2929f66d0489995e516e8d53e6561" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44535 } }
I20260620 14:52:36.319109 30523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:36.309072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:57579 (local address 127.0.0.1:44535)
0620 14:52:36.309683 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:36.309709 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:36.309758 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:52:36.310789 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:36.310820 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:36.311276 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:36.311666 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:36.312105 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:36.313327 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:36.314673 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:36.315329 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:36.315480 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:36.316563 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:36.316569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:36.316586 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:36.317107 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:36.317206 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:36.317245 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:36.317419 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:36.317713 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 14:52:36.318010 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":193,"threads_started":1}
I20260620 14:52:36.319564 30520 leader_election.cc:302] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14d2929f66d0489995e516e8d53e6561; no voters:
I20260620 14:52:36.319943 30520 leader_election.cc:288] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:36.320115 30525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:36.320410 30525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561 [term 1 LEADER]: Becoming Leader. State: Replica: 14d2929f66d0489995e516e8d53e6561, State: Running, Role: LEADER
I20260620 14:52:36.320858 30525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d2929f66d0489995e516e8d53e6561" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44535 } }
I20260620 14:52:36.322132 30520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:36.322780 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:36.323272 30527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14d2929f66d0489995e516e8d53e6561. Latest consensus state: current_term: 1 leader_uuid: "14d2929f66d0489995e516e8d53e6561" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d2929f66d0489995e516e8d53e6561" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44535 } } }
I20260620 14:52:36.323280 30526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14d2929f66d0489995e516e8d53e6561" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d2929f66d0489995e516e8d53e6561" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44535 } } }
I20260620 14:52:36.323454 30526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:36.323454 30527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:36.323964 30534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:36.328553 30534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:36.330703 30523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:36.325323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:48541 (local address 127.0.0.1:44535)
0620 14:52:36.325468 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:36.325474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:36.325618 (+ 144us) server_negotiation.cc:407] Connection header received
0620 14:52:36.325766 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:36.325771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:36.325846 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:36.325919 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:36.326336 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:36.327178 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:36.328386 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:36.328756 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:36.328860 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:36.330112 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:36.330116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:36.330119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:36.330131 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:36.330165 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:36.330176 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:36.330281 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:36.330501 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 14:52:36.330541 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:52:36.336084 30534 catalog_manager.cc:1347] Generated new cluster ID: 12592ecae2a74dc7910022e63b523532
I20260620 14:52:36.336158 30534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:36.346578 30534 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:36.347663 30534 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:36.352033 30534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14d2929f66d0489995e516e8d53e6561: Generated new TSK 0
I20260620 14:52:36.352864 30534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:36.361084 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 30452
W20260620 14:52:36.366819 30437 connection.cc:535] client connection to 127.0.0.1:44535 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44535): Connection reset by peer (error 104)
2026-06-20T14:52:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1456 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:36 dist-test-slave-65tz krb5kdc[30551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:36 dist-test-slave-65tz krb5kdc[30551](info): set up 2 sockets
Jun 20 14:52:36 dist-test-slave-65tz krb5kdc[30551](info): commencing operation
krb5kdc: starting...
W20260620 14:52:37.438366 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:37 dist-test-slave-65tz krb5kdc[30551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:37.573084 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41357
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:37.682227 30564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:37.682544 30564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:37.682619 30564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:37.686664 30564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:37.686772 30564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:37.686810 30564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:37.686849 30564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:37.686889 30564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:37.692261 30564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41357
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:37.693363 30564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:37.694409 30564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:37.704262 30572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:37.704231 30569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:37.704665 30564 server_base.cc:1034] running on GCE node
W20260620 14:52:37.704231 30570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:37.705310 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:37.706548 30564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:37.707741 30564 hybrid_clock.cc:648] HybridClock initialized: now 1781967157707717 us; error 42 us; skew 500 ppm
Jun 20 14:52:37 dist-test-slave-65tz krb5kdc[30551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:37.711051 30564 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:37.712105 30564 webserver.cc:460] Webserver started at http://127.0.0.1:40349/ using document root <none> and password file <none>
I20260620 14:52:37.712708 30564 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:37.712776 30564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:37.712999 30564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:37.715193 30564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "a317482a72184a35bab5451aa5a26cae"
format_stamp: "Formatted at 2026-06-20 14:52:37 on dist-test-slave-65tz"
server_key: "5d3cd802614ccecbabaf5e0f808626fe"
server_key_iv: "1db7ef77b28ca2faa559fd5b8fa84faa"
server_key_version: "encryptionkey@0"
I20260620 14:52:37.715797 30564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "a317482a72184a35bab5451aa5a26cae"
format_stamp: "Formatted at 2026-06-20 14:52:37 on dist-test-slave-65tz"
server_key: "5d3cd802614ccecbabaf5e0f808626fe"
server_key_iv: "1db7ef77b28ca2faa559fd5b8fa84faa"
server_key_version: "encryptionkey@0"
I20260620 14:52:37.719901 30564 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:52:37.722591 30579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:37.724181 30564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:37.724351 30564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "a317482a72184a35bab5451aa5a26cae"
format_stamp: "Formatted at 2026-06-20 14:52:37 on dist-test-slave-65tz"
server_key: "5d3cd802614ccecbabaf5e0f808626fe"
server_key_iv: "1db7ef77b28ca2faa559fd5b8fa84faa"
server_key_version: "encryptionkey@0"
I20260620 14:52:37.724469 30564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:37.729272 30564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:37.732630 30564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:37.732865 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:37.740736 30564 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41357
I20260620 14:52:37.740871 30631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41357 every 8 connection(s)
I20260620 14:52:37.741882 30564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:37.745714 30632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:37.750753 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 30564
I20260620 14:52:37.750954 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:37.751283 9079 external_mini_cluster.cc:1428] Setting key 7716f2284b66e4e181857425aaac0cd4
I20260620 14:52:37.753248 30632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae: Bootstrap starting.
I20260620 14:52:37.756285 30632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:37.757403 30632 log.cc:824] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:37.758149 30635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:37.753219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:48755 (local address 127.0.0.1:41357)
0620 14:52:37.753841 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:37.753872 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:37.753913 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:52:37.754792 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:37.754812 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:37.755155 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:37.755509 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:37.756666 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:37.756671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:37.756683 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:37.756694 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:37.756754 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:37.756774 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:37.756895 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:37.757035 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 14:52:37.757221 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":226,"threads_started":1}
I20260620 14:52:37.759768 30632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae: No bootstrap required, opened a new log
I20260620 14:52:37.761348 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:37.764752 30632 raft_consensus.cc:348] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a317482a72184a35bab5451aa5a26cae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41357 } }
I20260620 14:52:37.765097 30632 raft_consensus.cc:374] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:37.765249 30632 raft_consensus.cc:729] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a317482a72184a35bab5451aa5a26cae, State: Initialized, Role: FOLLOWER
I20260620 14:52:37.765486 30635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:37.763596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.84:57133 (local address 127.0.0.1:41357)
0620 14:52:37.763754 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:37.763760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:37.763866 (+ 106us) server_negotiation.cc:407] Connection header received
0620 14:52:37.763957 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:37.763960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:37.764002 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:37.764103 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:37.764977 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:37.764981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:37.764984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:37.764997 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:37.765031 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:37.765047 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:37.765156 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:37.765281 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:52:37.765337 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:52:37.765903 30632 consensus_queue.cc:260] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a317482a72184a35bab5451aa5a26cae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41357 } }
I20260620 14:52:37.766073 30632 raft_consensus.cc:388] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:37.766176 30632 raft_consensus.cc:482] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:37.766325 30632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:37.767477 30632 raft_consensus.cc:504] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a317482a72184a35bab5451aa5a26cae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41357 } }
I20260620 14:52:37.767884 30632 leader_election.cc:302] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a317482a72184a35bab5451aa5a26cae; no voters:
I20260620 14:52:37.768275 30632 leader_election.cc:288] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:37.768443 30642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:37.768745 30642 raft_consensus.cc:686] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae [term 1 LEADER]: Becoming Leader. State: Replica: a317482a72184a35bab5451aa5a26cae, State: Running, Role: LEADER
I20260620 14:52:37.769169 30642 consensus_queue.cc:237] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a317482a72184a35bab5451aa5a26cae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41357 } }
I20260620 14:52:37.769877 30632 sys_catalog.cc:564] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:37.771364 30644 sys_catalog.cc:455] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae [sys.catalog]: SysCatalogTable state changed. Reason: New leader a317482a72184a35bab5451aa5a26cae. Latest consensus state: current_term: 1 leader_uuid: "a317482a72184a35bab5451aa5a26cae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a317482a72184a35bab5451aa5a26cae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41357 } } }
I20260620 14:52:37.771546 30644 sys_catalog.cc:458] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:37.772539 30651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:37.772650 30643 sys_catalog.cc:455] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a317482a72184a35bab5451aa5a26cae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a317482a72184a35bab5451aa5a26cae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41357 } } }
I20260620 14:52:37.772763 30643 sys_catalog.cc:458] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:37.775480 30651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:37.781034 30651 catalog_manager.cc:1347] Generated new cluster ID: c6ddf44ebade4d9fbe11c4035df84535
I20260620 14:52:37.781112 30651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:37.790319 30651 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:37.790797 30651 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:37.804474 30651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a317482a72184a35bab5451aa5a26cae: Generated new TSK 0
I20260620 14:52:37.805649 30651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:37.815343 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 30564
2026-06-20T14:52:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:37 dist-test-slave-65tz krb5kdc[30663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:37 dist-test-slave-65tz krb5kdc[30663](info): set up 2 sockets
Jun 20 14:52:37 dist-test-slave-65tz krb5kdc[30663](info): commencing operation
krb5kdc: starting...
W20260620 14:52:38.934656 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:39 dist-test-slave-65tz krb5kdc[30663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:52:39.072680 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:33733
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:42453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:33733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:39.200587 30676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:39.200943 30676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:39.200994 30676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:39.206194 30676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:39.206295 30676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:39.206319 30676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:39.206339 30676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:39.206357 30676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:39.212637 30676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:42453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:33733
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:33733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:39.214025 30676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:39.215330 30676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:39.225569 30681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:39.225580 30684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:39.225626 30682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:39.226125 30676 server_base.cc:1034] running on GCE node
I20260620 14:52:39.226975 30676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:39.228391 30676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:39.229597 30676 hybrid_clock.cc:648] HybridClock initialized: now 1781967159229580 us; error 37 us; skew 500 ppm
Jun 20 14:52:39 dist-test-slave-65tz krb5kdc[30663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967159, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:39.232928 30676 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:52:39.233987 30676 webserver.cc:460] Webserver started at http://127.8.221.254:43095/ using document root <none> and password file <none>
I20260620 14:52:39.234609 30676 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:39.234679 30676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:39.234941 30676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:39.237061 30676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "a4260c1d28264cdf9d3d1f4d6df44008"
format_stamp: "Formatted at 2026-06-20 14:52:39 on dist-test-slave-65tz"
server_key: "e94589cfc08e13748b2e796d1d915948"
server_key_iv: "d0762e04589b205706118e55e75a3ad8"
server_key_version: "encryptionkey@0"
I20260620 14:52:39.237636 30676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "a4260c1d28264cdf9d3d1f4d6df44008"
format_stamp: "Formatted at 2026-06-20 14:52:39 on dist-test-slave-65tz"
server_key: "e94589cfc08e13748b2e796d1d915948"
server_key_iv: "d0762e04589b205706118e55e75a3ad8"
server_key_version: "encryptionkey@0"
I20260620 14:52:39.241710 30676 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 14:52:39.244371 30691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:39.245965 30676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:39.246091 30676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "a4260c1d28264cdf9d3d1f4d6df44008"
format_stamp: "Formatted at 2026-06-20 14:52:39 on dist-test-slave-65tz"
server_key: "e94589cfc08e13748b2e796d1d915948"
server_key_iv: "d0762e04589b205706118e55e75a3ad8"
server_key_version: "encryptionkey@0"
I20260620 14:52:39.246227 30676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:39.267131 30676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:39.270344 30676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:39.270556 30676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:39.278309 30676 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:33733
I20260620 14:52:39.278365 30743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:33733 every 8 connection(s)
I20260620 14:52:39.279289 30676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:39.280320 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 30676
I20260620 14:52:39.280480 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:39.280767 9079 external_mini_cluster.cc:1428] Setting key c36fa3e5eaa4395ea104534737bb7362
I20260620 14:52:39.283231 30744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:39.291666 30744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008: Bootstrap starting.
Jun 20 14:52:39 dist-test-slave-65tz krb5kdc[30663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:52:39.295142 30744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:39.296087 30744 log.cc:824] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:39.298403 30747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:39.282480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47463 (local address 127.8.221.254:33733)
0620 14:52:39.282963 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:39.282985 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:39.283039 (+ 54us) server_negotiation.cc:407] Connection header received
0620 14:52:39.284024 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:39.284050 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:39.284544 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:39.284955 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:39.285751 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:39.287390 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:39.288396 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:39.289109 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:39.289483 (+ 374us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:39.292498 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:39.292505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:39.292532 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:39.292569 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:39.294728 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:39.295404 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:39.295412 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:39.295418 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:39.295601 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:39.295960 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:39.295966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:39.295972 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:39.296522 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:39.296786 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:39.297197 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 14:52:39.297484 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":122,"threads_started":1}
I20260620 14:52:39.298882 30744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008: No bootstrap required, opened a new log
I20260620 14:52:39.301573 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:39.303861 30744 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4260c1d28264cdf9d3d1f4d6df44008" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33733 } }
I20260620 14:52:39.304234 30744 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:39.304301 30744 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4260c1d28264cdf9d3d1f4d6df44008, State: Initialized, Role: FOLLOWER
I20260620 14:52:39.304951 30744 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4260c1d28264cdf9d3d1f4d6df44008" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33733 } }
I20260620 14:52:39.305153 30744 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:39.305240 30744 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:39.305368 30744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:39.306680 30744 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4260c1d28264cdf9d3d1f4d6df44008" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33733 } }
I20260620 14:52:39.307094 30744 leader_election.cc:302] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4260c1d28264cdf9d3d1f4d6df44008; no voters:
I20260620 14:52:39.307462 30744 leader_election.cc:288] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:39.307665 30754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:39.308087 30754 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008 [term 1 LEADER]: Becoming Leader. State: Replica: a4260c1d28264cdf9d3d1f4d6df44008, State: Running, Role: LEADER
I20260620 14:52:39.308566 30754 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4260c1d28264cdf9d3d1f4d6df44008" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33733 } }
I20260620 14:52:39.309130 30744 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:39.310825 30755 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4260c1d28264cdf9d3d1f4d6df44008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4260c1d28264cdf9d3d1f4d6df44008" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33733 } } }
I20260620 14:52:39.310984 30755 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:39.311286 30755 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4260c1d28264cdf9d3d1f4d6df44008. Latest consensus state: current_term: 1 leader_uuid: "a4260c1d28264cdf9d3d1f4d6df44008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4260c1d28264cdf9d3d1f4d6df44008" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 33733 } } }
I20260620 14:52:39.311391 30755 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:39.311491 30763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:39.314822 30747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:39.304289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42093 (local address 127.8.221.254:33733)
0620 14:52:39.304429 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:39.304436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:39.304672 (+ 236us) server_negotiation.cc:407] Connection header received
0620 14:52:39.304771 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:39.304773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:39.304814 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:39.304893 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:39.305839 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:39.306411 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:39.307639 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:39.307995 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:39.308088 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:39.310147 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:39.310152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:39.310156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:39.310187 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:39.312163 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:39.312785 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:39.312790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:39.312792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:39.312917 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:39.313424 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:39.313428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:39.313430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:39.313667 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:39.313887 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:39.314538 (+ 651us) server_negotiation.cc:299] Negotiation successful
0620 14:52:39.314653 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:52:39.315953 30763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:39.322515 30763 catalog_manager.cc:1347] Generated new cluster ID: c705313d0ed24697987fa1b3d70cfc9e
I20260620 14:52:39.322594 30763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:39.361627 30763 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:39.363083 30763 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:39.371917 30763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4260c1d28264cdf9d3d1f4d6df44008: Generated new TSK 0
I20260620 14:52:39.372713 30763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:39.385262 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 30676
W20260620 14:52:39.392441 30661 connection.cc:535] client connection to 127.8.221.254:33733 recv error: Network error: failed to read from TLS socket (remote: 127.8.221.254:33733): Connection reset by peer (error 104)
2026-06-20T14:52:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1570 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:39 dist-test-slave-65tz krb5kdc[30775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:39 dist-test-slave-65tz krb5kdc[30775](info): set up 2 sockets
Jun 20 14:52:39 dist-test-slave-65tz krb5kdc[30775](info): commencing operation
krb5kdc: starting...
W20260620 14:52:40.483182 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:40 dist-test-slave-65tz krb5kdc[30775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:52:40.621562 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:34863
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:45625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:34863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:40.736115 30788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:40.736449 30788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:40.736524 30788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:40.740651 30788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:40.740767 30788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:40.740805 30788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:40.740844 30788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:40.740875 30788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:40.746631 30788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:45625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:34863
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:34863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:40.747782 30788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:40.748865 30788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:40.758267 30794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:40.758276 30793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:40.758409 30788 server_base.cc:1034] running on GCE node
W20260620 14:52:40.758699 30796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:40.759469 30788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:40.760857 30788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:40.762085 30788 hybrid_clock.cc:648] HybridClock initialized: now 1781967160762053 us; error 61 us; skew 500 ppm
Jun 20 14:52:40 dist-test-slave-65tz krb5kdc[30775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967160, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:40.768527 30788 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:52:40.769618 30788 webserver.cc:460] Webserver started at http://127.8.221.254:35559/ using document root <none> and password file <none>
I20260620 14:52:40.770354 30788 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:40.770452 30788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:40.770718 30788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:40.773118 30788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "3e38f8d3805b4a3f8648ec9de2504da2"
format_stamp: "Formatted at 2026-06-20 14:52:40 on dist-test-slave-65tz"
server_key: "0478a291a109ecb1125a788bfb6d15d5"
server_key_iv: "da3632b63d9529d0fdcb76bf65477563"
server_key_version: "encryptionkey@0"
I20260620 14:52:40.773767 30788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "3e38f8d3805b4a3f8648ec9de2504da2"
format_stamp: "Formatted at 2026-06-20 14:52:40 on dist-test-slave-65tz"
server_key: "0478a291a109ecb1125a788bfb6d15d5"
server_key_iv: "da3632b63d9529d0fdcb76bf65477563"
server_key_version: "encryptionkey@0"
I20260620 14:52:40.778345 30788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:52:40.789024 30803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:40.791112 30788 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.000s
I20260620 14:52:40.791285 30788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "3e38f8d3805b4a3f8648ec9de2504da2"
format_stamp: "Formatted at 2026-06-20 14:52:40 on dist-test-slave-65tz"
server_key: "0478a291a109ecb1125a788bfb6d15d5"
server_key_iv: "da3632b63d9529d0fdcb76bf65477563"
server_key_version: "encryptionkey@0"
I20260620 14:52:40.791433 30788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:40.819979 30788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:40.823894 30788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:40.824090 30788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:40.833359 30788 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:34863
I20260620 14:52:40.833384 30855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:34863 every 8 connection(s)
I20260620 14:52:40.834626 30788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:40.837642 30856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:40.839823 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 30788
I20260620 14:52:40.840019 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:40.840382 9079 external_mini_cluster.cc:1428] Setting key 2e5288bb8b23c69b387052a1d1473fff
I20260620 14:52:40.843814 30856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2: Bootstrap starting.
I20260620 14:52:40.846279 30856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:40.847113 30856 log.cc:824] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:40.849123 30856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2: No bootstrap required, opened a new log
I20260620 14:52:40.851418 30859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:40.842227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47135 (local address 127.8.221.254:34863)
0620 14:52:40.842748 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:40.842767 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:40.842811 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:52:40.843755 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:40.843777 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:40.844244 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:40.844582 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:40.844993 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:40.846299 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:40.847510 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:40.848060 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:40.848263 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:40.849359 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:40.849365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:40.849380 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:40.849948 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:40.850011 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:40.850038 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:40.850198 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:40.850416 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 14:52:40.850575 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":160,"threads_started":1}
I20260620 14:52:40.853148 30856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e38f8d3805b4a3f8648ec9de2504da2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 34863 } }
I20260620 14:52:40.853394 30856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:40.853474 30856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e38f8d3805b4a3f8648ec9de2504da2, State: Initialized, Role: FOLLOWER
I20260620 14:52:40.853997 30856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e38f8d3805b4a3f8648ec9de2504da2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 34863 } }
I20260620 14:52:40.854171 30856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:40.854230 30856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:40.854336 30856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:40.855180 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:40.855262 30856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e38f8d3805b4a3f8648ec9de2504da2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 34863 } }
I20260620 14:52:40.855592 30856 leader_election.cc:302] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e38f8d3805b4a3f8648ec9de2504da2; no voters:
I20260620 14:52:40.855924 30856 leader_election.cc:288] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:40.856102 30861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:40.856488 30861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2 [term 1 LEADER]: Becoming Leader. State: Replica: 3e38f8d3805b4a3f8648ec9de2504da2, State: Running, Role: LEADER
I20260620 14:52:40.856940 30861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e38f8d3805b4a3f8648ec9de2504da2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 34863 } }
I20260620 14:52:40.857282 30856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:40.860474 30863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e38f8d3805b4a3f8648ec9de2504da2. Latest consensus state: current_term: 1 leader_uuid: "3e38f8d3805b4a3f8648ec9de2504da2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e38f8d3805b4a3f8648ec9de2504da2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 34863 } } }
I20260620 14:52:40.860788 30863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:40.861030 30862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e38f8d3805b4a3f8648ec9de2504da2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e38f8d3805b4a3f8648ec9de2504da2" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 34863 } } }
I20260620 14:52:40.861135 30862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:40.863843 30859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:40.857709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59309 (local address 127.8.221.254:34863)
0620 14:52:40.857873 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:40.857878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:40.858516 (+ 638us) server_negotiation.cc:407] Connection header received
0620 14:52:40.858752 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:40.858755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:40.858809 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:40.858866 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:40.859312 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:40.860555 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:40.861694 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:40.862096 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:40.862248 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:40.863254 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:40.863259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:40.863262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:40.863278 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:40.863311 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:40.863324 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:40.863433 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:40.863634 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 14:52:40.863678 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 14:52:40.864631 30880 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:52:40.864728 30880 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:52:40.864820 30882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:40.865732 30882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:40.872009 30882 catalog_manager.cc:1347] Generated new cluster ID: 6aeb14a510474ef5942789a8a70be419
I20260620 14:52:40.872105 30882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:40.888826 30882 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:40.890987 30882 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:40.900168 30882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e38f8d3805b4a3f8648ec9de2504da2: Generated new TSK 0
I20260620 14:52:40.901083 30882 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:40.915611 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 30788
W20260620 14:52:40.921192 30773 connection.cc:535] client connection to 127.8.221.254:34863 recv error: Network error: failed to read from TLS socket (remote: 127.8.221.254:34863): Connection reset by peer (error 104)
2026-06-20T14:52:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1527 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:40 dist-test-slave-65tz krb5kdc[30887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:40 dist-test-slave-65tz krb5kdc[30887](info): set up 2 sockets
Jun 20 14:52:40 dist-test-slave-65tz krb5kdc[30887](info): commencing operation
krb5kdc: starting...
W20260620 14:52:42.002815 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:42 dist-test-slave-65tz krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:52:42.142323 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:37133
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:39993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:37133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:42.270375 30900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:42.270661 30900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:42.270705 30900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:42.274756 30900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:42.274811 30900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:42.274829 30900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:42.274847 30900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:42.274916 30900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:42.280265 30900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:39993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:37133
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:37133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:42.281491 30900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:42.282694 30900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:42.292606 30908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:42.292596 30905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:42.292724 30900 server_base.cc:1034] running on GCE node
W20260620 14:52:42.292596 30906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:42.293869 30900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:42.295313 30900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:42.296623 30900 hybrid_clock.cc:648] HybridClock initialized: now 1781967162296607 us; error 36 us; skew 500 ppm
Jun 20 14:52:42 dist-test-slave-65tz krb5kdc[30887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967162, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:42.299862 30900 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:52:42.301007 30900 webserver.cc:460] Webserver started at http://127.8.221.254:35109/ using document root <none> and password file <none>
I20260620 14:52:42.301694 30900 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:42.301791 30900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:42.302088 30900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:42.304515 30900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "05a80c44e1cb43de8060b35183190b34"
format_stamp: "Formatted at 2026-06-20 14:52:42 on dist-test-slave-65tz"
server_key: "1ae133f84a30336510dc1543bda7b376"
server_key_iv: "4d201a6dee84b9876454c837bea35db5"
server_key_version: "encryptionkey@0"
I20260620 14:52:42.305125 30900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "05a80c44e1cb43de8060b35183190b34"
format_stamp: "Formatted at 2026-06-20 14:52:42 on dist-test-slave-65tz"
server_key: "1ae133f84a30336510dc1543bda7b376"
server_key_iv: "4d201a6dee84b9876454c837bea35db5"
server_key_version: "encryptionkey@0"
I20260620 14:52:42.309484 30900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 14:52:42.312310 30915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:42.313758 30900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:52:42.313946 30900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "05a80c44e1cb43de8060b35183190b34"
format_stamp: "Formatted at 2026-06-20 14:52:42 on dist-test-slave-65tz"
server_key: "1ae133f84a30336510dc1543bda7b376"
server_key_iv: "4d201a6dee84b9876454c837bea35db5"
server_key_version: "encryptionkey@0"
I20260620 14:52:42.314065 30900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:42.319070 30900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:42.322280 30900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:42.322500 30900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:42.331076 30900 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:37133
I20260620 14:52:42.331103 30967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:37133 every 8 connection(s)
I20260620 14:52:42.332144 30900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:42.335132 30968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:42.339877 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 30900
I20260620 14:52:42.340040 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:42.340368 9079 external_mini_cluster.cc:1428] Setting key 30cb19d2601a194f3af63f69978d995c
I20260620 14:52:42.341436 30968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34: Bootstrap starting.
I20260620 14:52:42.343819 30968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:42.344653 30968 log.cc:824] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:42.346578 30971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:42.342032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34875 (local address 127.8.221.254:37133)
0620 14:52:42.342578 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:42.342598 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:42.342643 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:52:42.343447 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:42.343468 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:42.343825 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:42.344262 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:42.345146 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:42.345151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:42.345164 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:42.345174 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:42.345235 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:42.345255 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:42.345387 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:42.345472 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 14:52:42.345652 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":180,"threads_started":1}
I20260620 14:52:42.346853 30968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34: No bootstrap required, opened a new log
I20260620 14:52:42.349987 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:42.351017 30968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05a80c44e1cb43de8060b35183190b34" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 37133 } }
I20260620 14:52:42.351289 30968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:42.351354 30968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05a80c44e1cb43de8060b35183190b34, State: Initialized, Role: FOLLOWER
I20260620 14:52:42.351851 30968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05a80c44e1cb43de8060b35183190b34" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 37133 } }
I20260620 14:52:42.351976 30968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:42.352065 30968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:42.352201 30968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:42.353181 30968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05a80c44e1cb43de8060b35183190b34" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 37133 } }
I20260620 14:52:42.353525 30968 leader_election.cc:302] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05a80c44e1cb43de8060b35183190b34; no voters:
I20260620 14:52:42.353885 30968 leader_election.cc:288] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:42.353978 30978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:42.354189 30978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34 [term 1 LEADER]: Becoming Leader. State: Replica: 05a80c44e1cb43de8060b35183190b34, State: Running, Role: LEADER
I20260620 14:52:42.354418 30971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:42.352422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36155 (local address 127.8.221.254:37133)
0620 14:52:42.352591 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:42.352596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:42.352815 (+ 219us) server_negotiation.cc:407] Connection header received
0620 14:52:42.352958 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:42.352961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:42.353004 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:42.353089 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:42.353891 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:42.353894 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:42.353905 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:42.353916 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:42.353949 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:42.353965 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:42.354051 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:42.354213 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 14:52:42.354260 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 14:52:42.354480 30978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05a80c44e1cb43de8060b35183190b34" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 37133 } }
I20260620 14:52:42.355227 30968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:42.355767 30979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05a80c44e1cb43de8060b35183190b34. Latest consensus state: current_term: 1 leader_uuid: "05a80c44e1cb43de8060b35183190b34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05a80c44e1cb43de8060b35183190b34" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 37133 } } }
I20260620 14:52:42.355835 30980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05a80c44e1cb43de8060b35183190b34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05a80c44e1cb43de8060b35183190b34" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 37133 } } }
I20260620 14:52:42.355942 30979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:42.355947 30980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34 [sys.catalog]: This master's current role is: LEADER
W20260620 14:52:42.360244 30993 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:52:42.360339 30993 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:52:42.360425 30984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:42.361475 30984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:42.367456 30984 catalog_manager.cc:1347] Generated new cluster ID: 19982132eda24756a9e0f74a0af01d1d
I20260620 14:52:42.367535 30984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:42.387480 30984 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:42.387890 30984 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:42.395973 30984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05a80c44e1cb43de8060b35183190b34: Generated new TSK 0
I20260620 14:52:42.396749 30984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:42.404131 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 30900
2026-06-20T14:52:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1489 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:42 dist-test-slave-65tz krb5kdc[30999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:42 dist-test-slave-65tz krb5kdc[30999](info): set up 2 sockets
Jun 20 14:52:42 dist-test-slave-65tz krb5kdc[30999](info): commencing operation
krb5kdc: starting...
W20260620 14:52:43.466996 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:43 dist-test-slave-65tz krb5kdc[30999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:52:43.612453 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:36659
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:35503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:36659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:43.728456 31012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:43.728797 31012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:43.728873 31012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:43.732923 31012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:43.733018 31012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:43.733057 31012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:43.733111 31012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:43.733150 31012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:43.738539 31012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:35503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:36659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:36659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:43.739696 31012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:43.740767 31012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:43.750056 31017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:43.750154 31020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:43.750180 31012 server_base.cc:1034] running on GCE node
W20260620 14:52:43.750049 31018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:43.751317 31012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:43.752878 31012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:43.754070 31012 hybrid_clock.cc:648] HybridClock initialized: now 1781967163753993 us; error 99 us; skew 500 ppm
Jun 20 14:52:43 dist-test-slave-65tz krb5kdc[30999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967163, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:43.757642 31012 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:52:43.758946 31012 webserver.cc:460] Webserver started at http://127.8.221.254:35139/ using document root <none> and password file <none>
I20260620 14:52:43.759768 31012 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:43.759876 31012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:43.760160 31012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:43.762972 31012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "c402d48de1a043eb8f57c5f33d01c82e"
format_stamp: "Formatted at 2026-06-20 14:52:43 on dist-test-slave-65tz"
server_key: "c776737d7eef7217e3261130805e5789"
server_key_iv: "f6ed8af90f6e966fba11bd72c8961b01"
server_key_version: "encryptionkey@0"
I20260620 14:52:43.763698 31012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "c402d48de1a043eb8f57c5f33d01c82e"
format_stamp: "Formatted at 2026-06-20 14:52:43 on dist-test-slave-65tz"
server_key: "c776737d7eef7217e3261130805e5789"
server_key_iv: "f6ed8af90f6e966fba11bd72c8961b01"
server_key_version: "encryptionkey@0"
I20260620 14:52:43.768817 31012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:52:43.772392 31027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:43.774292 31012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 14:52:43.774533 31012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "c402d48de1a043eb8f57c5f33d01c82e"
format_stamp: "Formatted at 2026-06-20 14:52:43 on dist-test-slave-65tz"
server_key: "c776737d7eef7217e3261130805e5789"
server_key_iv: "f6ed8af90f6e966fba11bd72c8961b01"
server_key_version: "encryptionkey@0"
I20260620 14:52:43.774688 31012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:43.789768 31012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:43.794080 31012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:43.794346 31012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:43.803298 31012 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:36659
I20260620 14:52:43.803380 31079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:36659 every 8 connection(s)
I20260620 14:52:43.804562 31012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:43.808022 31080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:43.810525 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 31012
I20260620 14:52:43.810700 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:43.811072 9079 external_mini_cluster.cc:1428] Setting key ed5c595754c5583dc90c3b1aaa747da3
I20260620 14:52:43.815018 31080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e: Bootstrap starting.
I20260620 14:52:43.817587 31080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:43.818509 31080 log.cc:824] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:43.820840 31080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e: No bootstrap required, opened a new log
Jun 20 14:52:43 dist-test-slave-65tz krb5kdc[30999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:52:43.824263 31080 raft_consensus.cc:348] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c402d48de1a043eb8f57c5f33d01c82e" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 36659 } }
I20260620 14:52:43.824568 31080 raft_consensus.cc:374] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:43.824676 31080 raft_consensus.cc:729] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c402d48de1a043eb8f57c5f33d01c82e, State: Initialized, Role: FOLLOWER
I20260620 14:52:43.825223 31080 consensus_queue.cc:260] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c402d48de1a043eb8f57c5f33d01c82e" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 36659 } }
I20260620 14:52:43.825450 31080 raft_consensus.cc:388] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:43.825567 31080 raft_consensus.cc:482] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:43.825691 31080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:43.827373 31080 raft_consensus.cc:504] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c402d48de1a043eb8f57c5f33d01c82e" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 36659 } }
I20260620 14:52:43.827766 31080 leader_election.cc:302] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c402d48de1a043eb8f57c5f33d01c82e; no voters:
I20260620 14:52:43.828186 31080 leader_election.cc:288] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:43.828325 31085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:43.828492 31083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:43.812808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51371 (local address 127.8.221.254:36659)
0620 14:52:43.813308 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:43.813343 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:43.813554 (+ 211us) server_negotiation.cc:407] Connection header received
0620 14:52:43.814546 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:43.814572 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:43.815006 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:43.815439 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:43.816316 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:43.817461 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:43.818526 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:43.819137 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:43.819353 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:43.822165 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:43.822171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:43.822188 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:43.822218 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:43.824830 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:43.825310 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:43.825317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:43.825324 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:43.825499 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:43.825778 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:43.825783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:43.825785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:43.826350 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:43.826593 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:43.826870 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 14:52:43.827246 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":143,"threads_started":1}
I20260620 14:52:43.828608 31085 raft_consensus.cc:686] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e [term 1 LEADER]: Becoming Leader. State: Replica: c402d48de1a043eb8f57c5f33d01c82e, State: Running, Role: LEADER
I20260620 14:52:43.829006 31085 consensus_queue.cc:237] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c402d48de1a043eb8f57c5f33d01c82e" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 36659 } }
I20260620 14:52:43.829684 31080 sys_catalog.cc:564] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:43.832065 31087 sys_catalog.cc:455] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c402d48de1a043eb8f57c5f33d01c82e. Latest consensus state: current_term: 1 leader_uuid: "c402d48de1a043eb8f57c5f33d01c82e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c402d48de1a043eb8f57c5f33d01c82e" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 36659 } } }
I20260620 14:52:43.832257 31087 sys_catalog.cc:458] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:43.832648 31086 sys_catalog.cc:455] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c402d48de1a043eb8f57c5f33d01c82e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c402d48de1a043eb8f57c5f33d01c82e" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 36659 } } }
I20260620 14:52:43.832772 31086 sys_catalog.cc:458] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:43.832937 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:43.833091 31099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:43.837204 31099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:43.844940 31099 catalog_manager.cc:1347] Generated new cluster ID: dc34533d9fe3450fb828bc489590a859
I20260620 14:52:43.845062 31099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:43.846728 31083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:43.835643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33731 (local address 127.8.221.254:36659)
0620 14:52:43.835798 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:43.835805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:43.836077 (+ 272us) server_negotiation.cc:407] Connection header received
0620 14:52:43.836265 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:43.836270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:43.836332 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:43.836441 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:43.837248 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:43.838019 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:43.838961 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:43.839257 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:43.839391 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:43.841000 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:43.841005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:43.841008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:43.841047 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:43.843421 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:43.843938 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:43.843942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:43.843945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:43.844046 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:43.844376 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:43.844382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:43.844384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:43.844683 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:43.844876 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:43.846326 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0620 14:52:43.846536 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:52:43.855837 31099 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:43.857352 31099 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:43.863727 31099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c402d48de1a043eb8f57c5f33d01c82e: Generated new TSK 0
I20260620 14:52:43.864598 31099 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:43.871059 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 31012
2026-06-20T14:52:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1467 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:43 dist-test-slave-65tz krb5kdc[31111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:43 dist-test-slave-65tz krb5kdc[31111](info): set up 2 sockets
Jun 20 14:52:43 dist-test-slave-65tz krb5kdc[31111](info): commencing operation
krb5kdc: starting...
W20260620 14:52:45.007037 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:45 dist-test-slave-65tz krb5kdc[31111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:52:45.147046 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:43445
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:43123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:43445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:45.274605 31124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:45.274956 31124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:45.275040 31124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:45.279253 31124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:45.279349 31124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:45.279386 31124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:45.279444 31124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:45.279480 31124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:45.285223 31124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:43123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:43445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:43445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:45.286657 31124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:45.287916 31124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:45.298000 31130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:45.298043 31129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:45.298043 31132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:45.298518 31124 server_base.cc:1034] running on GCE node
I20260620 14:52:45.299223 31124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:45.300652 31124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:45.301908 31124 hybrid_clock.cc:648] HybridClock initialized: now 1781967165301875 us; error 47 us; skew 500 ppm
Jun 20 14:52:45 dist-test-slave-65tz krb5kdc[31111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967165, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:45.305474 31124 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:52:45.306746 31124 webserver.cc:460] Webserver started at http://127.8.221.254:33669/ using document root <none> and password file <none>
I20260620 14:52:45.307421 31124 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:45.307495 31124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:45.307744 31124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:45.310374 31124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "82248997d7394d588662edd95f162a46"
format_stamp: "Formatted at 2026-06-20 14:52:45 on dist-test-slave-65tz"
server_key: "fe7664efcff43bf6e18bfffcc4f825e0"
server_key_iv: "40a7420c0cf3939a3e5c13aef46c8743"
server_key_version: "encryptionkey@0"
I20260620 14:52:45.311038 31124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "82248997d7394d588662edd95f162a46"
format_stamp: "Formatted at 2026-06-20 14:52:45 on dist-test-slave-65tz"
server_key: "fe7664efcff43bf6e18bfffcc4f825e0"
server_key_iv: "40a7420c0cf3939a3e5c13aef46c8743"
server_key_version: "encryptionkey@0"
I20260620 14:52:45.315896 31124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:52:45.319171 31139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:45.320822 31124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:52:45.320958 31124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "82248997d7394d588662edd95f162a46"
format_stamp: "Formatted at 2026-06-20 14:52:45 on dist-test-slave-65tz"
server_key: "fe7664efcff43bf6e18bfffcc4f825e0"
server_key_iv: "40a7420c0cf3939a3e5c13aef46c8743"
server_key_version: "encryptionkey@0"
I20260620 14:52:45.321077 31124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:45.335914 31124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:45.339532 31124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:45.339709 31124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:45.349006 31124 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:43445
I20260620 14:52:45.349032 31191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:43445 every 8 connection(s)
I20260620 14:52:45.350317 31124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:45.353505 31192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:45.354790 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 31124
I20260620 14:52:45.354957 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:45.355281 9079 external_mini_cluster.cc:1428] Setting key d45c4ec5e5de11dccba1d5d6eed20fca
I20260620 14:52:45.360718 31192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46: Bootstrap starting.
I20260620 14:52:45.363070 31192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:45.363947 31192 log.cc:824] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:45.366351 31192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46: No bootstrap required, opened a new log
I20260620 14:52:45.366490 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:45.357242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36455 (local address 127.8.221.254:43445)
0620 14:52:45.357836 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:45.357890 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:45.357949 (+ 59us) server_negotiation.cc:407] Connection header received
0620 14:52:45.358843 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:45.358864 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:45.359309 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:45.359718 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:45.360204 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:45.361357 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:45.362381 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:45.362921 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:45.363105 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:45.364127 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:45.364136 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:45.364159 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:45.364835 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:45.364915 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:45.364956 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:45.365082 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:45.365205 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 14:52:45.365470 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":158,"threads_started":1}
I20260620 14:52:45.369961 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:45.370563 31192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82248997d7394d588662edd95f162a46" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43445 } }
I20260620 14:52:45.370823 31192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:45.370901 31192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82248997d7394d588662edd95f162a46, State: Initialized, Role: FOLLOWER
I20260620 14:52:45.371345 31192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82248997d7394d588662edd95f162a46" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43445 } }
I20260620 14:52:45.371522 31192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:45.371624 31192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:45.371759 31192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:45.372910 31192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82248997d7394d588662edd95f162a46" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43445 } }
I20260620 14:52:45.373288 31192 leader_election.cc:302] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82248997d7394d588662edd95f162a46; no voters:
I20260620 14:52:45.373643 31192 leader_election.cc:288] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:45.373793 31202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:45.374104 31202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46 [term 1 LEADER]: Becoming Leader. State: Replica: 82248997d7394d588662edd95f162a46, State: Running, Role: LEADER
I20260620 14:52:45.374572 31202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82248997d7394d588662edd95f162a46" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43445 } }
I20260620 14:52:45.375049 31192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:45.376489 31203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82248997d7394d588662edd95f162a46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82248997d7394d588662edd95f162a46" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43445 } } }
I20260620 14:52:45.376700 31203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:45.377002 31203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82248997d7394d588662edd95f162a46. Latest consensus state: current_term: 1 leader_uuid: "82248997d7394d588662edd95f162a46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82248997d7394d588662edd95f162a46" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43445 } } }
I20260620 14:52:45.377148 31203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:45.377671 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:45.372384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34795 (local address 127.8.221.254:43445)
0620 14:52:45.372538 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:45.372546 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:45.372676 (+ 130us) server_negotiation.cc:407] Connection header received
0620 14:52:45.372768 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:45.372774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:45.372841 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:45.372955 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:45.373284 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:45.374113 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:45.375309 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:45.375654 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:45.375760 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:45.376941 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:45.376947 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:45.376950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:45.376964 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:45.376996 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:45.377013 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:45.377089 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:45.377447 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 14:52:45.377502 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 14:52:45.380612 31217 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:52:45.380686 31217 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:52:45.380781 31213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:45.381910 31213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:45.388633 31213 catalog_manager.cc:1347] Generated new cluster ID: 53b709af1766433f872a8f55b6798935
I20260620 14:52:45.388715 31213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:45.406994 31213 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:45.407948 31213 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:45.417145 31213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82248997d7394d588662edd95f162a46: Generated new TSK 0
I20260620 14:52:45.417908 31213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:45.426017 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 31124
2026-06-20T14:52:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1554 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:45 dist-test-slave-65tz krb5kdc[31223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:45 dist-test-slave-65tz krb5kdc[31223](info): set up 2 sockets
Jun 20 14:52:45 dist-test-slave-65tz krb5kdc[31223](info): commencing operation
krb5kdc: starting...
W20260620 14:52:46.482501 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:46 dist-test-slave-65tz krb5kdc[31223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:52:46.621493 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:43831
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:34197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:43831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:46.738339 31236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:46.738662 31236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:46.738737 31236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:46.742859 31236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:46.742954 31236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:46.742990 31236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:46.743045 31236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:46.743093 31236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:46.748728 31236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:34197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:43831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:43831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:46.749926 31236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:46.750985 31236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:46.761008 31241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:46.761029 31242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:46.761145 31236 server_base.cc:1034] running on GCE node
W20260620 14:52:46.761390 31244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:46.762084 31236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:46.763270 31236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:46.764448 31236 hybrid_clock.cc:648] HybridClock initialized: now 1781967166764419 us; error 45 us; skew 500 ppm
Jun 20 14:52:46 dist-test-slave-65tz krb5kdc[31223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967166, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:46.771579 31236 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:52:46.772707 31236 webserver.cc:460] Webserver started at http://127.8.221.254:33861/ using document root <none> and password file <none>
I20260620 14:52:46.773363 31236 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:46.773465 31236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:46.773718 31236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:46.776109 31236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "16d973f11ecb48cbb43cb9cf0985f740"
format_stamp: "Formatted at 2026-06-20 14:52:46 on dist-test-slave-65tz"
server_key: "1c0896ed4d88dde5aff4fb05c1ef9ed7"
server_key_iv: "acd93f9fc9a9940ca217c9ec21e583bc"
server_key_version: "encryptionkey@0"
I20260620 14:52:46.776814 31236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "16d973f11ecb48cbb43cb9cf0985f740"
format_stamp: "Formatted at 2026-06-20 14:52:46 on dist-test-slave-65tz"
server_key: "1c0896ed4d88dde5aff4fb05c1ef9ed7"
server_key_iv: "acd93f9fc9a9940ca217c9ec21e583bc"
server_key_version: "encryptionkey@0"
I20260620 14:52:46.782935 31236 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 14:52:46.786936 31251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:46.788995 31236 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:52:46.789156 31236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "16d973f11ecb48cbb43cb9cf0985f740"
format_stamp: "Formatted at 2026-06-20 14:52:46 on dist-test-slave-65tz"
server_key: "1c0896ed4d88dde5aff4fb05c1ef9ed7"
server_key_iv: "acd93f9fc9a9940ca217c9ec21e583bc"
server_key_version: "encryptionkey@0"
I20260620 14:52:46.789294 31236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:46.794533 31236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:46.798038 31236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:46.798251 31236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:46.806366 31236 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:43831
I20260620 14:52:46.806398 31303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:43831 every 8 connection(s)
I20260620 14:52:46.807416 31236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:46.810075 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 31236
I20260620 14:52:46.810253 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:46.810613 9079 external_mini_cluster.cc:1428] Setting key 3622bcc767a2f7cf85ded12febc5b4fd
I20260620 14:52:46.811161 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:46.817494 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:46.812351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56523 (local address 127.8.221.254:43831)
0620 14:52:46.812946 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:46.812964 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:46.813222 (+ 258us) server_negotiation.cc:407] Connection header received
0620 14:52:46.814595 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:46.814616 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:46.814929 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:46.815253 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:46.816223 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:46.816228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:46.816239 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:46.816249 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:46.816305 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:46.816323 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:46.816456 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:46.816543 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 14:52:46.816713 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":207,"threads_started":1}
I20260620 14:52:46.819958 31304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740: Bootstrap starting.
I20260620 14:52:46.820955 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:46.823323 31304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:46.824365 31304 log.cc:824] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:46.825903 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:46.823819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59913 (local address 127.8.221.254:43831)
0620 14:52:46.823985 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:46.823990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:46.824237 (+ 247us) server_negotiation.cc:407] Connection header received
0620 14:52:46.824396 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:46.824398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:46.824431 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:46.824521 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:46.825290 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:46.825293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:46.825296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:46.825307 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:46.825337 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:46.825348 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:46.825431 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:46.825616 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 14:52:46.825704 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:52:46.827037 31304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740: No bootstrap required, opened a new log
I20260620 14:52:46.830932 31304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d973f11ecb48cbb43cb9cf0985f740" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43831 } }
I20260620 14:52:46.831178 31304 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:46.831212 31304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d973f11ecb48cbb43cb9cf0985f740, State: Initialized, Role: FOLLOWER
I20260620 14:52:46.831647 31304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d973f11ecb48cbb43cb9cf0985f740" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43831 } }
I20260620 14:52:46.831773 31304 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:46.831815 31304 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:46.831892 31304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:46.832852 31304 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d973f11ecb48cbb43cb9cf0985f740" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43831 } }
I20260620 14:52:46.833195 31304 leader_election.cc:302] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16d973f11ecb48cbb43cb9cf0985f740; no voters:
I20260620 14:52:46.833482 31304 leader_election.cc:288] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:46.833722 31314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:46.834090 31314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740 [term 1 LEADER]: Becoming Leader. State: Replica: 16d973f11ecb48cbb43cb9cf0985f740, State: Running, Role: LEADER
I20260620 14:52:46.834501 31314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d973f11ecb48cbb43cb9cf0985f740" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43831 } }
I20260620 14:52:46.835142 31304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:46.836526 31315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16d973f11ecb48cbb43cb9cf0985f740" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d973f11ecb48cbb43cb9cf0985f740" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43831 } } }
I20260620 14:52:46.836669 31315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:46.836937 31316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16d973f11ecb48cbb43cb9cf0985f740. Latest consensus state: current_term: 1 leader_uuid: "16d973f11ecb48cbb43cb9cf0985f740" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d973f11ecb48cbb43cb9cf0985f740" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 43831 } } }
I20260620 14:52:46.837102 31316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740 [sys.catalog]: This master's current role is: LEADER
W20260620 14:52:46.840094 31329 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:52:46.840173 31329 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:52:46.840260 31328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:46.841032 31328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:46.846205 31328 catalog_manager.cc:1347] Generated new cluster ID: d12989402ee7434fa4dedaafd6005ea0
I20260620 14:52:46.846282 31328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:46.858634 31328 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:46.859019 31328 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:46.874042 31328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16d973f11ecb48cbb43cb9cf0985f740: Generated new TSK 0
I20260620 14:52:46.874797 31328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:46.879509 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 31236
2026-06-20T14:52:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1453 ms)
[----------] 18 tests from AuthTokenIssuingTest (26876 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:46 dist-test-slave-65tz krb5kdc[31335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:46 dist-test-slave-65tz krb5kdc[31335](info): set up 2 sockets
Jun 20 14:52:46 dist-test-slave-65tz krb5kdc[31335](info): commencing operation
krb5kdc: starting...
W20260620 14:52:47.938434 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:48 dist-test-slave-65tz krb5kdc[31335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:52:48.073935 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38235
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:43921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:38235,127.8.221.253:42437,127.8.221.252:35051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:48.185827 31348 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:52:48.186137 31348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:48.186173 31348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:48.186275 31348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:48.190471 31348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:48.190529 31348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:48.190549 31348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:48.190567 31348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:48.190582 31348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:48.195802 31348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:43921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:38235,127.8.221.253:42437,127.8.221.252:35051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:48.196830 31348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:48.197827 31348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:48.206704 31354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:48.206722 31353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:48.206704 31356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:48.207171 31348 server_base.cc:1034] running on GCE node
I20260620 14:52:48.207881 31348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:48.209251 31348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:48.210464 31348 hybrid_clock.cc:648] HybridClock initialized: now 1781967168210422 us; error 54 us; skew 500 ppm
Jun 20 14:52:48 dist-test-slave-65tz krb5kdc[31335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:48.213606 31348 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:52:48.214758 31348 webserver.cc:460] Webserver started at http://127.8.221.254:35561/ using document root <none> and password file <none>
I20260620 14:52:48.215394 31348 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:48.215457 31348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:48.215653 31348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:48.218025 31348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "daa7603de2534a7b9a20c013f3d4e7b4"
format_stamp: "Formatted at 2026-06-20 14:52:48 on dist-test-slave-65tz"
server_key: "c055caf11827b63424cf082deb9c2096"
server_key_iv: "803c1c3ed7e2499e1287ba6122afa6ae"
server_key_version: "encryptionkey@0"
I20260620 14:52:48.218605 31348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "daa7603de2534a7b9a20c013f3d4e7b4"
format_stamp: "Formatted at 2026-06-20 14:52:48 on dist-test-slave-65tz"
server_key: "c055caf11827b63424cf082deb9c2096"
server_key_iv: "803c1c3ed7e2499e1287ba6122afa6ae"
server_key_version: "encryptionkey@0"
I20260620 14:52:48.222894 31348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:52:48.225914 31363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:48.227401 31348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:48.227520 31348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "daa7603de2534a7b9a20c013f3d4e7b4"
format_stamp: "Formatted at 2026-06-20 14:52:48 on dist-test-slave-65tz"
server_key: "c055caf11827b63424cf082deb9c2096"
server_key_iv: "803c1c3ed7e2499e1287ba6122afa6ae"
server_key_version: "encryptionkey@0"
I20260620 14:52:48.227624 31348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:48.242349 31348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:48.245582 31348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:48.245765 31348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:48.254133 31348 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:38235
I20260620 14:52:48.254456 31415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:38235 every 8 connection(s)
I20260620 14:52:48.255363 31348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:48.258639 31416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:48.260587 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 31348
I20260620 14:52:48.260741 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:48.261059 9079 external_mini_cluster.cc:1428] Setting key ea7fe0db320d9c1e0ee52207c1b60abc
I20260620 14:52:48.263922 31416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.221.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.253@KRBTEST.COM" created.
Jun 20 14:52:48 dist-test-slave-65tz krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:52:48.283716 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.266680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45142 (local address 127.8.221.254:38235)
0620 14:52:48.267302 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.267320 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.267685 (+ 365us) server_negotiation.cc:407] Connection header received
0620 14:52:48.269519 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.269524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.269706 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.269956 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.271296 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.272613 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.273769 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.274514 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.274743 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.278240 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:48.278246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:48.278255 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:48.278277 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:48.280165 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.280873 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.280878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.280880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.280994 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.281421 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.281424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.281425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.281871 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:48.281991 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:48.282540 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 14:52:48.282779 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":480,"thread_start_us":169,"threads_started":1}
I20260620 14:52:48.283720 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.266680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38235 (local address 127.0.0.1:45142)
0620 14:52:48.267424 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.267509 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:48.269143 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:48.269979 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:48.269989 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:48.270557 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:48.271099 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:48.271121 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.272756 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.272760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:48.273640 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:48.273650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.274764 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.274771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:48.275047 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.275761 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:48.275787 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:48.278051 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:48.280306 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:48.280320 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:48.280337 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:48.280730 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:48.281114 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:48.281119 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:48.281124 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:48.281292 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:48.282004 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:48.282013 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:48.282235 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 14:52:48.282467 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 14:52:48.282859 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":91,"spinlock_wait_cycles":52992,"thread_start_us":169,"threads_started":1}
I20260620 14:52:48.286794 31416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:52:48.287755 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.287413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:42437 (local address 127.0.0.1:52726)
0620 14:52:48.287563 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.287655 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:42437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 14:52:48.288029 31364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.221.253:42437: connect: Connection refused (error 111)
I20260620 14:52:48.289121 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.288843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:42437 (local address 127.0.0.1:52732)
0620 14:52:48.288953 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.289043 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:42437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 14:52:48.289309 31416 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.253:42437: Network error: Client connection negotiation failed: client connection to 127.8.221.253:42437: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.253.keytab.
Entry for principal kudu/127.8.221.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.253.keytab.
I20260620 14:52:48.293239 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.253:42437
--webserver_interface=127.8.221.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.253.keytab
--principal=kudu/127.8.221.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:43921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:38235,127.8.221.253:42437,127.8.221.252:35051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:52:48.338568 31416 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } attempt: 1
I20260620 14:52:48.339741 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.339355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:42437 (local address 127.0.0.1:52736)
0620 14:52:48.339577 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.339656 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:42437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 14:52:48.340545 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.340342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:42437 (local address 127.0.0.1:52752)
0620 14:52:48.340444 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.340497 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:42437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 14:52:48.340672 31416 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.253:42437: Network error: Client connection negotiation failed: client connection to 127.8.221.253:42437: connect: Connection refused (error 111)
I20260620 14:52:48.408906 31416 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } attempt: 2
I20260620 14:52:48.409970 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.409461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:42437 (local address 127.0.0.1:52768)
0620 14:52:48.409700 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.409839 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:42437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 14:52:48.410827 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.410595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:42437 (local address 127.0.0.1:52774)
0620 14:52:48.410707 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.410766 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:42437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 14:52:48.411015 31416 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.253:42437: Network error: Client connection negotiation failed: client connection to 127.8.221.253:42437: connect: Connection refused (error 111)
W20260620 14:52:48.420969 31423 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:52:48.421326 31423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:48.421352 31423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:48.421489 31423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:48.426026 31423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:48.426105 31423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:48.426127 31423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:48.426146 31423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:48.426162 31423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:48.432133 31423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:43921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:38235,127.8.221.253:42437,127.8.221.252:35051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.253.keytab
--principal=kudu/127.8.221.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.253:42437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.221.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:48.433525 31423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:48.434865 31423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:48.444875 31428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:48.444875 31429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:48.444875 31431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:48.445223 31423 server_base.cc:1034] running on GCE node
I20260620 14:52:48.446022 31423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:48.447296 31423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:48.448467 31423 hybrid_clock.cc:648] HybridClock initialized: now 1781967168448452 us; error 36 us; skew 500 ppm
Jun 20 14:52:48 dist-test-slave-65tz krb5kdc[31335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:48.451576 31423 init.cc:377] Logged in from keytab as kudu/127.8.221.253@KRBTEST.COM (short username kudu)
I20260620 14:52:48.452546 31423 webserver.cc:460] Webserver started at http://127.8.221.253:34323/ using document root <none> and password file <none>
I20260620 14:52:48.453164 31423 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:48.453251 31423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:48.453495 31423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:48.455677 31423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-1/data/instance:
uuid: "2d120fa5e50c4a62a4a9020d5cfd322e"
format_stamp: "Formatted at 2026-06-20 14:52:48 on dist-test-slave-65tz"
server_key: "5ebcdf363e9d11605dd69a74a2adc58b"
server_key_iv: "94574596193fddb62f9b519d9cf8e3e8"
server_key_version: "encryptionkey@0"
I20260620 14:52:48.456259 31423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-1/wal/instance:
uuid: "2d120fa5e50c4a62a4a9020d5cfd322e"
format_stamp: "Formatted at 2026-06-20 14:52:48 on dist-test-slave-65tz"
server_key: "5ebcdf363e9d11605dd69a74a2adc58b"
server_key_iv: "94574596193fddb62f9b519d9cf8e3e8"
server_key_version: "encryptionkey@0"
I20260620 14:52:48.460425 31423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:52:48.463276 31438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:48.464834 31423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:52:48.464990 31423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-1/wal
uuid: "2d120fa5e50c4a62a4a9020d5cfd322e"
format_stamp: "Formatted at 2026-06-20 14:52:48 on dist-test-slave-65tz"
server_key: "5ebcdf363e9d11605dd69a74a2adc58b"
server_key_iv: "94574596193fddb62f9b519d9cf8e3e8"
server_key_version: "encryptionkey@0"
I20260620 14:52:48.465142 31423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:48.494307 31423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:48.497404 31423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:48.497607 31423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:48.502241 31416 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } attempt: 3
I20260620 14:52:48.503440 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.502983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:42437 (local address 127.0.0.1:52790)
0620 14:52:48.503210 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.503312 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:42437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 14:52:48.504393 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.504114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:42437 (local address 127.0.0.1:52806)
0620 14:52:48.504246 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.504311 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:42437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 14:52:48.504669 31416 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.253:42437: Network error: Client connection negotiation failed: client connection to 127.8.221.253:42437: connect: Connection refused (error 111)
I20260620 14:52:48.506062 31423 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.253:42437
I20260620 14:52:48.506345 31490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.253:42437 every 8 connection(s)
I20260620 14:52:48.507220 31423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-1/data/info.pb
I20260620 14:52:48.510702 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 31423
I20260620 14:52:48.510815 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-1/wal/instance
I20260620 14:52:48.510988 31491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:48.511188 9079 external_mini_cluster.cc:1428] Setting key 7496f51c14b73b4a77fcb05e8887efa1
I20260620 14:52:48.517890 31491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.221.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.252@KRBTEST.COM" created.
Jun 20 14:52:48 dist-test-slave-65tz krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.253@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:52:48.536077 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.520430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45152 (local address 127.8.221.254:38235)
0620 14:52:48.520638 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.520644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.521978 (+ 1334us) server_negotiation.cc:407] Connection header received
0620 14:52:48.523298 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.523301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.523372 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.523456 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.525131 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.526090 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.528054 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.528334 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.528442 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.532007 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:48.532011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:48.532013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:48.532037 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:48.533675 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.534312 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.534318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.534320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.534402 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.534774 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.534777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.534780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.534989 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:48.535107 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:48.535742 (+ 635us) server_negotiation.cc:299] Negotiation successful
0620 14:52:48.535883 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:52:48.537248 31494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.520857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38235 (local address 127.0.0.1:45152)
0620 14:52:48.521584 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.521656 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:48.522976 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:48.523744 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:48.523760 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:48.524352 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:48.524937 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:48.524953 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.526287 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.526293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:48.527842 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:48.527859 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.528520 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.528527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:48.528777 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.529560 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:48.529588 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:48.531796 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:48.533835 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:48.533843 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:48.533864 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:48.534215 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:48.534501 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:48.534505 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:48.534507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:48.534672 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:48.535102 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:48.535111 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:48.535458 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 14:52:48.535676 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 14:52:48.536104 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":170,"threads_started":1}
I20260620 14:52:48.539213 31491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.252.keytab.
Entry for principal kudu/127.8.221.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.252.keytab.
I20260620 14:52:48.541983 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.252:35051
--webserver_interface=127.8.221.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.252.keytab
--principal=kudu/127.8.221.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:43921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:38235,127.8.221.253:42437,127.8.221.252:35051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:52:48 dist-test-slave-65tz krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.253@KRBTEST.COM for kudu/127.8.221.253@KRBTEST.COM
I20260620 14:52:48.551529 31494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.539921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:42437 (local address 127.0.0.1:52812)
0620 14:52:48.540082 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.540118 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:48.540213 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:48.541241 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:48.541245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:48.541476 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:48.541598 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:48.541607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.542918 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.542922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:48.543686 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:48.543695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.544642 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.544645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:48.544733 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.545326 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:48.545338 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:48.546947 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:48.549614 (+ 2667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:48.549617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:48.549619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:48.549955 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:48.550319 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:48.550321 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:48.550322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:48.550407 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:48.551062 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:48.551065 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:48.551172 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 14:52:48.551270 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 14:52:48.551390 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 14:52:48.551913 31496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.540079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52812 (local address 127.8.221.253:42437)
0620 14:52:48.540504 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.540515 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.540565 (+ 50us) server_negotiation.cc:407] Connection header received
0620 14:52:48.540617 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.540621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.540833 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.541023 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.541758 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.542789 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.543816 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.544507 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.544634 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.547123 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:48.547138 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:48.547163 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:48.547201 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:48.549471 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.550087 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.550095 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.550098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.550208 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.550505 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.550510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.550515 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.550925 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:48.551113 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:48.551409 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 14:52:48.551619 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":178,"threads_started":1}
I20260620 14:52:48.554386 31491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:52:48.555276 31494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.554973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:35051 (local address 127.0.0.1:52344)
0620 14:52:48.555105 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.555190 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:35051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 14:52:48.555502 31440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.221.252:35051: connect: Connection refused (error 111)
I20260620 14:52:48.556373 31494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.556173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:35051 (local address 127.0.0.1:52354)
0620 14:52:48.556277 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.556324 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:35051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 14:52:48.556559 31491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.252:35051: Network error: Client connection negotiation failed: client connection to 127.8.221.252:35051: connect: Connection refused (error 111)
I20260620 14:52:48.605927 31491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } attempt: 1
I20260620 14:52:48.607054 31494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.606652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:35051 (local address 127.0.0.1:52366)
0620 14:52:48.606893 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.606972 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:35051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 14:52:48.607888 31494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.607678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:35051 (local address 127.0.0.1:52374)
0620 14:52:48.607774 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.607827 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:35051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 14:52:48.608017 31491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.252:35051: Network error: Client connection negotiation failed: client connection to 127.8.221.252:35051: connect: Connection refused (error 111)
I20260620 14:52:48.647889 31416 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } attempt: 4
Jun 20 14:52:48 dist-test-slave-65tz krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for kudu/127.8.221.253@KRBTEST.COM
I20260620 14:52:48.658910 31496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.648900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52820 (local address 127.8.221.253:42437)
0620 14:52:48.649108 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.649114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.649214 (+ 100us) server_negotiation.cc:407] Connection header received
0620 14:52:48.649374 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.649379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.649458 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.649572 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.650138 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.651604 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.652984 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.653251 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.653384 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.655786 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:48.655790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:48.655793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:48.655822 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:48.657247 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.657681 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.657686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.657688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.657761 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.658075 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.658080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.658081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.658266 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:48.658377 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:48.658635 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 14:52:48.658749 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:52:48.658910 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.648705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:42437 (local address 127.0.0.1:52820)
0620 14:52:48.649086 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.649110 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:48.649251 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:48.649603 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:48.649606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:48.649780 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:48.650011 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:48.650017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.651751 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.651754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:48.652753 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:48.652762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.653384 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.653388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:48.653492 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.654066 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:48.654078 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:48.655641 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:48.657383 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:48.657386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:48.657389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:48.657585 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:48.657899 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:48.657902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:48.657904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:48.657987 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:48.658366 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:48.658369 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:48.658481 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 14:52:48.658586 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 14:52:48.658735 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215}
I20260620 14:52:48.660184 31416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:52:48.660989 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.660755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:35051 (local address 127.0.0.1:52388)
0620 14:52:48.660886 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.660937 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:35051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 14:52:48.661604 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.661405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:35051 (local address 127.0.0.1:52402)
0620 14:52:48.661505 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.661552 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:35051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 14:52:48.661749 31416 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.252:35051: Network error: Client connection negotiation failed: client connection to 127.8.221.252:35051: connect: Connection refused (error 111)
W20260620 14:52:48.675192 31497 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:52:48.675559 31497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:48.675625 31497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:48.675696 31497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:52:48.676173 31491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } attempt: 2
I20260620 14:52:48.677011 31494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.676747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:35051 (local address 127.0.0.1:52410)
0620 14:52:48.676882 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.676942 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:35051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 14:52:48.677654 31494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.677459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:35051 (local address 127.0.0.1:52414)
0620 14:52:48.677555 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.677601 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:35051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 14:52:48.677775 31491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.252:35051: Network error: Client connection negotiation failed: client connection to 127.8.221.252:35051: connect: Connection refused (error 111)
W20260620 14:52:48.680085 31497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:48.680178 31497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:48.680215 31497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:48.680266 31497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:48.680300 31497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:48.685990 31497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:43921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:38235,127.8.221.253:42437,127.8.221.252:35051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.252.keytab
--principal=kudu/127.8.221.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.252:35051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.221.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:48.687425 31497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:48.688699 31497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:48.698912 31504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:48.698962 31503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:48.698973 31506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:48.699205 31497 server_base.cc:1034] running on GCE node
I20260620 14:52:48.700037 31497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:48.701421 31497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:48.702605 31497 hybrid_clock.cc:648] HybridClock initialized: now 1781967168702547 us; error 79 us; skew 500 ppm
Jun 20 14:52:48 dist-test-slave-65tz krb5kdc[31335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:48.706105 31497 init.cc:377] Logged in from keytab as kudu/127.8.221.252@KRBTEST.COM (short username kudu)
I20260620 14:52:48.707321 31497 webserver.cc:460] Webserver started at http://127.8.221.252:38127/ using document root <none> and password file <none>
I20260620 14:52:48.707993 31497 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:48.708072 31497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:48.708287 31497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:48.710922 31497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-2/data/instance:
uuid: "65b0e4c7fb5c4f18a7028c086e264db3"
format_stamp: "Formatted at 2026-06-20 14:52:48 on dist-test-slave-65tz"
server_key: "97e91b544467310327b6e411f9ac7872"
server_key_iv: "91e40b8eb6c6f3eab86bcd3a65c48f4a"
server_key_version: "encryptionkey@0"
I20260620 14:52:48.711560 31497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-2/wal/instance:
uuid: "65b0e4c7fb5c4f18a7028c086e264db3"
format_stamp: "Formatted at 2026-06-20 14:52:48 on dist-test-slave-65tz"
server_key: "97e91b544467310327b6e411f9ac7872"
server_key_iv: "91e40b8eb6c6f3eab86bcd3a65c48f4a"
server_key_version: "encryptionkey@0"
I20260620 14:52:48.716296 31497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:52:48.719584 31513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:48.720930 31416 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } attempt: 1
I20260620 14:52:48.721707 31497 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 14:52:48.721974 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.721458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:35051 (local address 127.0.0.1:52420)
0620 14:52:48.721702 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.721855 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:35051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 14:52:48.721906 31497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-2/wal
uuid: "65b0e4c7fb5c4f18a7028c086e264db3"
format_stamp: "Formatted at 2026-06-20 14:52:48 on dist-test-slave-65tz"
server_key: "97e91b544467310327b6e411f9ac7872"
server_key_iv: "91e40b8eb6c6f3eab86bcd3a65c48f4a"
server_key_version: "encryptionkey@0"
I20260620 14:52:48.722044 31497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:48.722851 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.722621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:35051 (local address 127.0.0.1:52432)
0620 14:52:48.722736 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.722796 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:35051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 14:52:48.723011 31416 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.252:35051: Network error: Client connection negotiation failed: client connection to 127.8.221.252:35051: connect: Connection refused (error 111)
I20260620 14:52:48.733156 31497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:48.736294 31497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:48.736500 31497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:48.743851 31497 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.252:35051
I20260620 14:52:48.744140 31565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.252:35051 every 8 connection(s)
I20260620 14:52:48.744926 31497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-2/data/info.pb
I20260620 14:52:48.748636 31566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:48.750269 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 31497
I20260620 14:52:48.750437 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966871461349-9079-0/minicluster-data/master-2/wal/instance
I20260620 14:52:48.750838 9079 external_mini_cluster.cc:1428] Setting key bdc3317e6e4d1b290d9cce3bd3865258
I20260620 14:52:48.754305 31566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:52:48 dist-test-slave-65tz krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.252@KRBTEST.COM
Jun 20 14:52:48 dist-test-slave-65tz krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.252@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:52:48.766654 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.756479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45156 (local address 127.8.221.254:38235)
0620 14:52:48.756707 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.756713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.757026 (+ 313us) server_negotiation.cc:407] Connection header received
0620 14:52:48.757251 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.757259 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.757310 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.757407 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.758284 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.758857 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.759844 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.760086 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.760203 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.763250 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:48.763253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:48.763255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:48.763281 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:48.764721 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.765210 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.765214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.765216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.765291 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.765560 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.765566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.765568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.765791 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:48.765933 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:48.766291 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 14:52:48.766439 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:52:48.767040 31569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.752629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52442 (local address 127.8.221.252:35051)
0620 14:52:48.753198 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.753216 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.753257 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:52:48.754159 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.754180 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.754566 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.754965 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.755836 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.757316 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.758239 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.758724 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.758940 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.761904 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:48.761909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:48.761920 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:48.761945 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:48.763867 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.764362 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.764368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.764374 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.764485 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.764755 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.764759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.764761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.765184 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:48.765412 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:48.765651 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 14:52:48.765988 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":180,"threads_started":1}
I20260620 14:52:48.767042 31570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.756361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38235 (local address 127.0.0.1:45156)
0620 14:52:48.756864 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.756913 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:48.757123 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:48.757467 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:48.757475 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:48.757978 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:48.758150 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:48.758157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.758990 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.758994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:48.759714 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:48.759723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.760252 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.760256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:48.760403 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.760990 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:48.761014 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:48.763054 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:48.764834 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:48.764845 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:48.764859 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:48.765095 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:48.765381 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:48.765386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:48.765387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:48.765475 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:48.765951 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:48.765958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:48.766072 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 14:52:48.766223 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 14:52:48.766439 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":198,"threads_started":1}
I20260620 14:52:48.768486 31566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:52:48.769030 31491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } attempt: 3
Jun 20 14:52:48 dist-test-slave-65tz krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.252@KRBTEST.COM for kudu/127.8.221.253@KRBTEST.COM
Jun 20 14:52:48 dist-test-slave-65tz krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.253@KRBTEST.COM for kudu/127.8.221.252@KRBTEST.COM
Jun 20 14:52:48 dist-test-slave-65tz krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:52:48.779316 31494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.769470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:35051 (local address 127.0.0.1:52456)
0620 14:52:48.769674 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.769689 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:48.769774 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:48.770191 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:48.770194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:48.770398 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:48.770564 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:48.770590 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.771606 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.771609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:48.772450 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:48.772459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.773353 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.773359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:48.773459 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.774035 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:48.774046 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:48.775448 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:48.777550 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:48.777555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:48.777557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:48.777949 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:48.778256 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:48.778259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:48.778261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:48.778405 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:48.778810 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:48.778812 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:48.778896 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 14:52:48.778978 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:52:48.779134 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"spinlock_wait_cycles":1280}
I20260620 14:52:48.779350 31570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.769090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:42437 (local address 127.0.0.1:52830)
0620 14:52:48.769258 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.769273 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:48.769347 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:48.769657 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:48.769661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:48.769886 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:48.770007 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:48.770013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.771123 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.771126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:48.771810 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:48.771816 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.772259 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.772263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:48.772356 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.772834 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:48.772848 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:48.775363 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:48.777514 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:48.777520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:48.777525 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:48.777718 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:48.778036 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:48.778040 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:48.778044 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:48.778143 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:48.778734 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:48.778740 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:48.778827 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 14:52:48.779086 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 14:52:48.779208 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":1}
I20260620 14:52:48.779707 31496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.769206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52830 (local address 127.8.221.253:42437)
0620 14:52:48.769338 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.769342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.769356 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:52:48.769497 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.769500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.769545 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.769632 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.770135 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.770991 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.771935 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.772158 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.772260 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.775499 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:48.775503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:48.775506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:48.775535 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:48.777388 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.777852 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.777856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.777858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.777940 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.778255 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.778258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.778260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.778595 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:48.778664 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:48.779028 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 14:52:48.779612 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":481,"server-negotiator.queue_time_us":58,"spinlock_wait_cycles":3456}
I20260620 14:52:48.780211 31569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.769582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52456 (local address 127.8.221.252:35051)
0620 14:52:48.769922 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.769926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.769940 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:52:48.769972 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.769975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.770009 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.770137 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.770751 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.771450 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.772834 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.773197 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.773671 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.775565 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:48.775571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:48.775577 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:48.775599 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:48.777438 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.778055 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.778058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.778059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.778154 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.778507 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.778511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.778512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.778710 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:48.779883 (+ 1173us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:48.779954 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 14:52:48.780058 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"spinlock_wait_cycles":768}
I20260620 14:52:48.780315 31566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:52:48.781301 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.770517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.8.221.254:38235)
0620 14:52:48.770673 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.770679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.770694 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:52:48.770733 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.770738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.770786 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.770872 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.771505 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.772292 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.773138 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.773390 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.773500 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.776311 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:48.776314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:48.776316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:48.776335 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:48.778923 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.779639 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.779644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.779646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.779748 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.780071 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.780076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.780078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.780324 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:48.780585 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:48.781054 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 14:52:48.781186 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:52:48.788226 31491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e: Bootstrap starting.
Jun 20 14:52:48 dist-test-slave-65tz krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.252@KRBTEST.COM for kudu/127.8.221.252@KRBTEST.COM
I20260620 14:52:48.790175 31416 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } attempt: 2
I20260620 14:52:48.791816 31491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e: Neither blocks nor log segments found. Creating new log.
Jun 20 14:52:48 dist-test-slave-65tz krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.253@KRBTEST.COM
I20260620 14:52:48.792701 31491 log.cc:824] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:48.793280 31569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.782765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.8.221.252:35051)
0620 14:52:48.783971 (+ 1206us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.783975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.783990 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:52:48.784025 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.784028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.784068 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.784149 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.784657 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.785465 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.786486 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.786801 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.786906 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.789453 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:48.789457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:48.789459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:48.789480 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:48.791433 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.791980 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.791985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.791987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.792127 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.792413 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.792416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.792417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.792673 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:48.792778 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:48.792992 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 14:52:48.793145 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":1134}
I20260620 14:52:48.793280 31570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.782698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:35051 (local address 127.0.0.1:52462)
0620 14:52:48.782871 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.782889 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:48.782979 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:48.784265 (+ 1286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:48.784268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:48.784445 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:48.784539 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:48.784547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.785593 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.785597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:48.786350 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:48.786361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.787020 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.787023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:48.787096 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.787698 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:48.787713 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:48.789314 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:48.791551 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:48.791554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:48.791556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:48.791797 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:48.792228 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:48.792234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:48.792238 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:48.792318 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:48.792766 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:48.792769 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:48.792854 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 14:52:48.792952 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 14:52:48.793090 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 14:52:48.795921 31491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e: No bootstrap required, opened a new log
I20260620 14:52:48.796656 31496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.784572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.8.221.253:42437)
0620 14:52:48.784680 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.784684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.785853 (+ 1169us) server_negotiation.cc:407] Connection header received
0620 14:52:48.786049 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.786052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.786102 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.786150 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.787056 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.787816 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.788863 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.789173 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.789245 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.792609 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:48.792613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:48.792616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:48.792643 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:48.794670 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.795149 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.795153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.795156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.795244 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.795577 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.795580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.795581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.795785 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:48.795962 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:48.796341 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 14:52:48.796491 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 14:52:48 dist-test-slave-65tz krb5kdc[31335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for kudu/127.8.221.252@KRBTEST.COM
I20260620 14:52:48.798492 31566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3: Bootstrap starting.
I20260620 14:52:48.800565 31491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } } peers { permanent_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } } peers { permanent_uuid: "65b0e4c7fb5c4f18a7028c086e264db3" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } }
I20260620 14:52:48.800877 31491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d120fa5e50c4a62a4a9020d5cfd322e, State: Initialized, Role: FOLLOWER
I20260620 14:52:48.801132 31566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:48.801288 31491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } } peers { permanent_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } } peers { permanent_uuid: "65b0e4c7fb5c4f18a7028c086e264db3" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } }
I20260620 14:52:48.801855 31573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } } peers { permanent_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } } peers { permanent_uuid: "65b0e4c7fb5c4f18a7028c086e264db3" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } } }
I20260620 14:52:48.801987 31573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:52:48.802069 31566 log.cc:824] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:48.802942 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.790668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:35051 (local address 127.0.0.1:52464)
0620 14:52:48.790960 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:48.790975 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:48.791055 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:48.793825 (+ 2770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:48.793830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:48.794041 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:48.794187 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:48.794199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.795291 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.795295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:48.796054 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:48.796064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.796675 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.796679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:48.796760 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.797234 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:48.797248 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:48.798828 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:48.800989 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:48.800992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:48.800995 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:48.801257 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:48.801652 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:48.801656 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:48.801658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:48.801775 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:48.802388 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:48.802393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:48.802526 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 14:52:48.802622 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:52:48.802788 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212}
I20260620 14:52:48.802968 31491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:48.803808 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.791204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52464 (local address 127.8.221.252:35051)
0620 14:52:48.793484 (+ 2280us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.793496 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.793512 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:52:48.793558 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.793563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.793627 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.794314 (+ 687us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.794429 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.795148 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.796212 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.796546 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.796977 (+ 431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.798996 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:48.799000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:48.799003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:48.799029 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:48.800846 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.801387 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.801390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.801392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.801489 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.801944 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.801948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.801950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.802189 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:48.802291 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:48.803259 (+ 968us) server_negotiation.cc:299] Negotiation successful
0620 14:52:48.803613 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2196,"thread_start_us":139,"threads_started":1}
I20260620 14:52:48.805032 31566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3: No bootstrap required, opened a new log
I20260620 14:52:48.806535 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:48.808274 31416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4: Bootstrap starting.
I20260620 14:52:48.808756 31566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } } peers { permanent_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } } peers { permanent_uuid: "65b0e4c7fb5c4f18a7028c086e264db3" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } }
I20260620 14:52:48.809000 31566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b0e4c7fb5c4f18a7028c086e264db3, State: Initialized, Role: FOLLOWER
I20260620 14:52:48.809326 31566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } } peers { permanent_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } } peers { permanent_uuid: "65b0e4c7fb5c4f18a7028c086e264db3" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } }
I20260620 14:52:48.808151 31390 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "daa7603de2534a7b9a20c013f3d4e7b4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45164
W20260620 14:52:48.810613 31585 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:52:48.810700 31585 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:52:48.810679 31566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:48.811249 31416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:48.812163 31416 log.cc:824] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:48.814882 31416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4: No bootstrap required, opened a new log
W20260620 14:52:48.816403 31599 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:52:48.816493 31599 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:52:48.817898 31416 raft_consensus.cc:348] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } } peers { permanent_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } } peers { permanent_uuid: "65b0e4c7fb5c4f18a7028c086e264db3" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } }
I20260620 14:52:48.818157 31416 raft_consensus.cc:729] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa7603de2534a7b9a20c013f3d4e7b4, State: Initialized, Role: FOLLOWER
I20260620 14:52:48.818523 31416 consensus_queue.cc:260] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } } peers { permanent_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } } peers { permanent_uuid: "65b0e4c7fb5c4f18a7028c086e264db3" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } }
I20260620 14:52:48.819262 31587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } } peers { permanent_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } } peers { permanent_uuid: "65b0e4c7fb5c4f18a7028c086e264db3" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } } }
I20260620 14:52:48.819402 31587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:52:48.820019 31416 sys_catalog.cc:564] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:48.819100 31423 master_runner.cc:186] Error getting master registration for 127.8.221.254:38235: OK, instance_id { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" instance_seqno: 1781967168251798 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:52:48.822701 31423 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:52:48.822978 31390 raft_consensus.cc:482] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:52:48.823140 31390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:48.819023 31497 master_runner.cc:186] Error getting master registration for 127.8.221.254:38235: OK, instance_id { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" instance_seqno: 1781967168251798 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:52:48.823549 31497 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:52:48.824672 31390 raft_consensus.cc:504] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } } peers { permanent_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } } peers { permanent_uuid: "65b0e4c7fb5c4f18a7028c086e264db3" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } }
I20260620 14:52:48.825831 31390 leader_election.cc:288] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [CANDIDATE]: Term 1 election: Requested vote from peers 2d120fa5e50c4a62a4a9020d5cfd322e (127.8.221.253:42437), 65b0e4c7fb5c4f18a7028c086e264db3 (127.8.221.252:35051)
I20260620 14:52:48.826766 31465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e"
I20260620 14:52:48.826833 31540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "65b0e4c7fb5c4f18a7028c086e264db3"
I20260620 14:52:48.827030 31540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:48.827030 31465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:48.828310 31540 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daa7603de2534a7b9a20c013f3d4e7b4 in term 1.
I20260620 14:52:48.828408 31465 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate daa7603de2534a7b9a20c013f3d4e7b4 in term 1.
I20260620 14:52:48.829028 31365 leader_election.cc:302] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b0e4c7fb5c4f18a7028c086e264db3, daa7603de2534a7b9a20c013f3d4e7b4; no voters:
I20260620 14:52:48.829528 31618 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } } peers { permanent_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } } peers { permanent_uuid: "65b0e4c7fb5c4f18a7028c086e264db3" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } } }
I20260620 14:52:48.829732 31618 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:52:48.829890 31618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [term 1 FOLLOWER]: Leader election won for term 1
W20260620 14:52:48.829979 31612 catalog_manager.cc:1558] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:52:48.830092 31612 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:52:48.830021 31618 raft_consensus.cc:686] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [term 1 LEADER]: Becoming Leader. State: Replica: daa7603de2534a7b9a20c013f3d4e7b4, State: Running, Role: LEADER
I20260620 14:52:48.830533 31618 consensus_queue.cc:237] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } } peers { permanent_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } } peers { permanent_uuid: "65b0e4c7fb5c4f18a7028c086e264db3" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } }
I20260620 14:52:48.832810 31618 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daa7603de2534a7b9a20c013f3d4e7b4. Latest consensus state: current_term: 1 leader_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } } peers { permanent_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } } peers { permanent_uuid: "65b0e4c7fb5c4f18a7028c086e264db3" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } } }
I20260620 14:52:48.832935 31618 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:48.833377 31620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:48.834589 31620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:48.834619 31348 master_runner.cc:263] No leader master found from master daa7603de2534a7b9a20c013f3d4e7b4
I20260620 14:52:48.834722 31348 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:52:48.854741 31540 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3 [term 1 FOLLOWER]: Refusing update from remote peer daa7603de2534a7b9a20c013f3d4e7b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:48.855013 31465 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e [term 1 FOLLOWER]: Refusing update from remote peer daa7603de2534a7b9a20c013f3d4e7b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:48.855578 31601 consensus_queue.cc:1035] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b0e4c7fb5c4f18a7028c086e264db3" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:48.856124 31601 consensus_queue.cc:1035] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:48.858227 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.838437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.8.221.252:35051)
0620 14:52:48.838591 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.838596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.840036 (+ 1440us) server_negotiation.cc:407] Connection header received
0620 14:52:48.840232 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.840235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.840285 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.840337 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.841994 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.842871 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.847080 (+ 4209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.847445 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.847518 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.853139 (+ 5621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:48.853143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:48.853147 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:48.853176 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:48.854909 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.856239 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.856243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.856245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.856328 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.857198 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.857201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.857203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.857435 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:48.857556 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:48.857939 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 14:52:48.858066 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:52:48.861042 31573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e [sys.catalog]: SysCatalogTable state changed. Reason: New leader daa7603de2534a7b9a20c013f3d4e7b4. Latest consensus state: current_term: 1 leader_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } } peers { permanent_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } } peers { permanent_uuid: "65b0e4c7fb5c4f18a7028c086e264db3" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } } }
I20260620 14:52:48.861228 31573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:52:48.864266 31496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.839467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52844 (local address 127.8.221.253:42437)
0620 14:52:48.839582 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.839586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.850240 (+ 10654us) server_negotiation.cc:407] Connection header received
0620 14:52:48.850492 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.850496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.850562 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.850623 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.851600 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.852437 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.856709 (+ 4272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.857061 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.857138 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.859127 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:48.859131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:48.859134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:48.859164 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:48.861377 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.862867 (+ 1490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.862871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.862873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.862974 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.863287 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.863291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.863292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.863529 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:48.863645 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:48.863995 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 14:52:48.864114 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:52:48.864404 31587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daa7603de2534a7b9a20c013f3d4e7b4. Latest consensus state: current_term: 1 leader_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } } peers { permanent_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } } peers { permanent_uuid: "65b0e4c7fb5c4f18a7028c086e264db3" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } } }
I20260620 14:52:48.864574 31587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:52:48.868908 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:48.838831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45168 (local address 127.8.221.254:38235)
0620 14:52:48.838946 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.838950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.841912 (+ 2962us) server_negotiation.cc:407] Connection header received
0620 14:52:48.844156 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.844159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.844208 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.844258 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.845149 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.845926 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.849891 (+ 3965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:48.850228 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:48.850304 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:48.858967 (+ 8663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:48.858971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:48.858974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:48.859003 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:48.862369 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.867539 (+ 5170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.867543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.867545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.867636 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:48.867921 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:48.867925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:48.867927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:48.868133 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:48.868240 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:48.868638 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 14:52:48.868754 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:52:48.876811 31601 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } } peers { permanent_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } } peers { permanent_uuid: "65b0e4c7fb5c4f18a7028c086e264db3" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } } }
I20260620 14:52:48.877010 31601 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:48.878015 31573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } } peers { permanent_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } } peers { permanent_uuid: "65b0e4c7fb5c4f18a7028c086e264db3" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } } }
I20260620 14:52:48.878225 31573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:52:48.878479 31625 mvcc.cc:205] Tried to move back new op lower bound from 7298937523618521088 to 7298937523534020608. Current Snapshot: MvccSnapshot[applied={T|T < 7298937523618521088}]
I20260620 14:52:48.878614 31620 catalog_manager.cc:1347] Generated new cluster ID: 5105d32c4b784349b86111cefdeda0b2
I20260620 14:52:48.878659 31620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:48.883044 31601 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } } peers { permanent_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } } peers { permanent_uuid: "65b0e4c7fb5c4f18a7028c086e264db3" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } } }
I20260620 14:52:48.883208 31601 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:48.884485 31587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daa7603de2534a7b9a20c013f3d4e7b4" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38235 } } peers { permanent_uuid: "2d120fa5e50c4a62a4a9020d5cfd322e" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 42437 } } peers { permanent_uuid: "65b0e4c7fb5c4f18a7028c086e264db3" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 35051 } } }
I20260620 14:52:48.884610 31587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:52:48.885347 31627 mvcc.cc:205] Tried to move back new op lower bound from 7298937523618521088 to 7298937523534020608. Current Snapshot: MvccSnapshot[applied={T|T < 7298937523618521088}]
I20260620 14:52:48.895323 31620 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:48.896624 31620 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:48.911445 31620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daa7603de2534a7b9a20c013f3d4e7b4: Generated new TSK 0
I20260620 14:52:48.912541 31620 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:52:48.945219 31571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:48.941860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52468 (local address 127.8.221.252:35051)
0620 14:52:48.942158 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.942163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.944056 (+ 1893us) server_negotiation.cc:407] Connection header received
0620 14:52:48.944250 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.944254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.944306 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.944359 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.945080 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52468: BlockingRecv error: recv got EOF from 127.0.0.1:52468 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 14:52:48.946794 31419 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:48.942840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45178 (local address 127.8.221.254:38235)
0620 14:52:48.942968 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.942972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.945892 (+ 2920us) server_negotiation.cc:407] Connection header received
0620 14:52:48.946071 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.946075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.946124 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.946175 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.946674 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45178: BlockingRecv error: recv got EOF from 127.0.0.1:45178 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 14:52:48.948247 31496 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:48.943473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.8.221.253:42437)
0620 14:52:48.943586 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.943590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.947365 (+ 3775us) server_negotiation.cc:407] Connection header received
0620 14:52:48.947534 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.947538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.947600 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.947654 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.948137 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52858: BlockingRecv error: recv got EOF from 127.0.0.1:52858 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 14:52:48.952864 31496 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:48.951398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.8.221.253:42437)
0620 14:52:48.951500 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.951505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.951923 (+ 418us) server_negotiation.cc:407] Connection header received
0620 14:52:48.952150 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.952153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.952230 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.952280 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.952765 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52870: BlockingRecv error: recv got EOF from 127.0.0.1:52870 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 14:52:48.957990 31496 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:48.956525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.8.221.253:42437)
0620 14:52:48.956622 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.956626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.957090 (+ 464us) server_negotiation.cc:407] Connection header received
0620 14:52:48.957264 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.957267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.957315 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.957364 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.957892 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52880: BlockingRecv error: recv got EOF from 127.0.0.1:52880 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 14:52:48.963804 31496 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:48.962358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.8.221.253:42437)
0620 14:52:48.962456 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.962461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.962917 (+ 456us) server_negotiation.cc:407] Connection header received
0620 14:52:48.963112 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.963115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.963164 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.963215 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.963695 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52884: BlockingRecv error: recv got EOF from 127.0.0.1:52884 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 14:52:48.971488 31496 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:48.970086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.8.221.253:42437)
0620 14:52:48.970218 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.970222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.970669 (+ 447us) server_negotiation.cc:407] Connection header received
0620 14:52:48.970840 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.970843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.970903 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.970952 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.971395 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52892: BlockingRecv error: recv got EOF from 127.0.0.1:52892 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 14:52:48.984191 31496 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:48.982464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.8.221.253:42437)
0620 14:52:48.982586 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:48.982590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:48.983177 (+ 587us) server_negotiation.cc:407] Connection header received
0620 14:52:48.983405 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:48.983409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:48.983465 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:48.983522 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:48.984082 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52894: BlockingRecv error: recv got EOF from 127.0.0.1:52894 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 14:52:49.005638 31496 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:49.003178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.8.221.253:42437)
0620 14:52:49.003381 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:49.003387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:49.004401 (+ 1014us) server_negotiation.cc:407] Connection header received
0620 14:52:49.004647 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:49.004651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:49.004712 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:49.004774 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:49.005509 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52906: BlockingRecv error: recv got EOF from 127.0.0.1:52906 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
W20260620 14:52:49.042057 31496 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:49.040607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.8.221.253:42437)
0620 14:52:49.040754 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:49.040759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:49.041023 (+ 264us) server_negotiation.cc:407] Connection header received
0620 14:52:49.041185 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:49.041189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:49.041240 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:49.041357 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:49.041939 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52908: BlockingRecv error: recv got EOF from 127.0.0.1:52908 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 14:52:49.109843 31496 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:49.108260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52916 (local address 127.8.221.253:42437)
0620 14:52:49.108528 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:49.108533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:49.108774 (+ 241us) server_negotiation.cc:407] Connection header received
0620 14:52:49.108864 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:49.108867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:49.108936 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:49.109037 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:49.109567 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52916: BlockingRecv error: recv got EOF from 127.0.0.1:52916 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
W20260620 14:52:49.241763 31496 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:49.240194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.8.221.253:42437)
0620 14:52:49.240364 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:49.240367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:49.240596 (+ 229us) server_negotiation.cc:407] Connection header received
0620 14:52:49.240783 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:49.240786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:49.240878 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:49.241001 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:49.241614 (+ 613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52930: BlockingRecv error: recv got EOF from 127.0.0.1:52930 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 14:52:49.501874 31496 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:49.500321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52934 (local address 127.8.221.253:42437)
0620 14:52:49.500520 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:49.500527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:49.500740 (+ 213us) server_negotiation.cc:407] Connection header received
0620 14:52:49.500901 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:49.500904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:49.500971 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:49.501068 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:49.501603 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52934: BlockingRecv error: recv got EOF from 127.0.0.1:52934 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 14:52:49.812247 31585 catalog_manager.cc:1259] Loaded cluster ID: 5105d32c4b784349b86111cefdeda0b2
I20260620 14:52:49.812335 31585 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e: loading cluster ID for follower catalog manager: success
I20260620 14:52:49.814036 31585 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e: acquiring CA information for follower catalog manager: success
I20260620 14:52:49.815196 31585 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2d120fa5e50c4a62a4a9020d5cfd322e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:52:49.818218 31599 catalog_manager.cc:1259] Loaded cluster ID: 5105d32c4b784349b86111cefdeda0b2
I20260620 14:52:49.818317 31599 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3: loading cluster ID for follower catalog manager: success
I20260620 14:52:49.820497 31599 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3: acquiring CA information for follower catalog manager: success
I20260620 14:52:49.822221 31599 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65b0e4c7fb5c4f18a7028c086e264db3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:52:50.021145 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:50.016219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52946 (local address 127.8.221.253:42437)
0620 14:52:50.016542 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:50.016547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:50.016566 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:52:50.016744 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:50.016747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:50.016917 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:50.017043 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:50.017355 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:50.018073 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:50.018977 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:50.019226 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:50.019321 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:50.019823 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:50.020481 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:50.020593 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:50.020701 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 14:52:50.020887 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":88,"threads_started":1}
I20260620 14:52:50.029115 31705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:50.024751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52470 (local address 127.8.221.252:35051)
0620 14:52:50.025069 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:50.025074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:50.025090 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:52:50.025330 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:50.025333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:50.025488 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:50.025626 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:50.025888 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:50.026512 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:50.027461 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:50.027696 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:50.027795 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:50.028071 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:50.028636 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:50.028745 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:50.028808 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 14:52:50.028868 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":87,"threads_started":1}
I20260620 14:52:50.038075 31705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:50.033256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52482 (local address 127.8.221.252:35051)
0620 14:52:50.033401 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:50.033408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:50.033863 (+ 455us) server_negotiation.cc:407] Connection header received
0620 14:52:50.034118 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:50.034124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:50.034196 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:50.034284 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:50.034674 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:50.035486 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:50.036717 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:50.037001 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:50.037113 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:50.037455 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:50.037558 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:50.037627 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:50.037784 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 14:52:50.037895 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:52:50.038519 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:50.033710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.8.221.253:42437)
0620 14:52:50.033864 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:50.033870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:50.034003 (+ 133us) server_negotiation.cc:407] Connection header received
0620 14:52:50.034175 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:50.034178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:50.034234 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:50.034283 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:50.035025 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:50.035582 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:50.037056 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:50.037348 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:50.037472 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:50.037900 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:50.038013 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:50.038109 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:50.038269 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 14:52:50.038345 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:52:50.039064 31711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:50.033142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45182 (local address 127.8.221.254:38235)
0620 14:52:50.033593 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:50.033598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:50.033616 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:52:50.033754 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:50.033762 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:50.034046 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:50.034492 (+ 446us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:50.034959 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:50.035743 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:50.037103 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:50.037502 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:50.037628 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:50.038050 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:50.038629 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:50.038737 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:50.038823 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 14:52:50.038895 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":59,"threads_started":1}
I20260620 14:52:50.043174 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 31348
I20260620 14:52:50.050988 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 31423
I20260620 14:52:50.058458 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 31497
2026-06-20T14:52:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3180 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:50 dist-test-slave-65tz krb5kdc[31716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:50 dist-test-slave-65tz krb5kdc[31716](info): set up 2 sockets
Jun 20 14:52:50 dist-test-slave-65tz krb5kdc[31716](info): commencing operation
krb5kdc: starting...
W20260620 14:52:51.138993 9079 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/test-user.keytab.
Jun 20 14:52:51 dist-test-slave-65tz krb5kdc[31716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.221.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
Entry for principal kudu/127.8.221.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab.
I20260620 14:52:51.270337 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38287
--webserver_interface=127.8.221.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:38287,127.8.221.253:34763,127.8.221.252:37833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:51.407420 31729 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:52:51.407857 31729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:51.407922 31729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:51.408028 31729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:51.412622 31729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:51.412757 31729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:51.412801 31729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:51.412845 31729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:51.412882 31729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:51.419170 31729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:38287,127.8.221.253:34763,127.8.221.252:37833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.254.keytab
--principal=kudu/127.8.221.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.254:38287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.221.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:51.420658 31729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:51.422034 31729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:51.431936 31737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:51.431946 31735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:51.432118 31729 server_base.cc:1034] running on GCE node
W20260620 14:52:51.432291 31734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:51.432956 31729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:51.434293 31729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:51.435482 31729 hybrid_clock.cc:648] HybridClock initialized: now 1781967171435452 us; error 53 us; skew 500 ppm
Jun 20 14:52:51 dist-test-slave-65tz krb5kdc[31716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967171, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:51.438516 31729 init.cc:377] Logged in from keytab as kudu/127.8.221.254@KRBTEST.COM (short username kudu)
I20260620 14:52:51.439435 31729 webserver.cc:460] Webserver started at http://127.8.221.254:46215/ using document root <none> and password file <none>
I20260620 14:52:51.440035 31729 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:51.440158 31729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:51.440400 31729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:51.442561 31729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-0/data/instance:
uuid: "3fcd21ceab624a4ebd3884a81ae4f50d"
format_stamp: "Formatted at 2026-06-20 14:52:51 on dist-test-slave-65tz"
server_key: "0c471ada0619d3d15cf8fc8614252816"
server_key_iv: "43525f9dd26c637bdad1ddb946c9c42a"
server_key_version: "encryptionkey@0"
I20260620 14:52:51.443123 31729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-0/wal/instance:
uuid: "3fcd21ceab624a4ebd3884a81ae4f50d"
format_stamp: "Formatted at 2026-06-20 14:52:51 on dist-test-slave-65tz"
server_key: "0c471ada0619d3d15cf8fc8614252816"
server_key_iv: "43525f9dd26c637bdad1ddb946c9c42a"
server_key_version: "encryptionkey@0"
I20260620 14:52:51.447126 31729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:52:51.449944 31744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:51.451407 31729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:51.451551 31729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-0/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-0/wal
uuid: "3fcd21ceab624a4ebd3884a81ae4f50d"
format_stamp: "Formatted at 2026-06-20 14:52:51 on dist-test-slave-65tz"
server_key: "0c471ada0619d3d15cf8fc8614252816"
server_key_iv: "43525f9dd26c637bdad1ddb946c9c42a"
server_key_version: "encryptionkey@0"
I20260620 14:52:51.451660 31729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:51.471927 31729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:51.475337 31729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:51.475545 31729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:51.483444 31729 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.254:38287
I20260620 14:52:51.483794 31796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.254:38287 every 8 connection(s)
I20260620 14:52:51.484460 31729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:51.487918 31797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:51.488922 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 31729
I20260620 14:52:51.489056 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-0/wal/instance
I20260620 14:52:51.489418 9079 external_mini_cluster.cc:1428] Setting key 266d30f02c33f9fb76d2d6ac3e0f023c
I20260620 14:52:51.493454 31797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.221.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.253@KRBTEST.COM" created.
Jun 20 14:52:51 dist-test-slave-65tz krb5kdc[31716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967171, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:52:51.512107 31800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.496389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52270 (local address 127.8.221.254:38287)
0620 14:52:51.496890 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:51.496911 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:51.497276 (+ 365us) server_negotiation.cc:407] Connection header received
0620 14:52:51.498570 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:51.498577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:51.498807 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:51.498991 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:51.499962 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.501100 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.502756 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.503198 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.503411 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.505996 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:51.506006 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:51.506025 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:51.506057 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:51.508310 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.508918 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.508925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.508927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.509095 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.509470 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.509475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.509479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.510092 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:51.510256 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:51.510775 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 14:52:51.511084 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":167,"threads_started":1}
I20260620 14:52:51.512130 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.496392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38287 (local address 127.0.0.1:52270)
0620 14:52:51.497104 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.497145 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:51.498156 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:51.499032 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:51.499037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:51.499492 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:51.499801 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:51.499815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.501269 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.501273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:51.502608 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:51.502624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.503393 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.503398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:51.503519 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.504133 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:51.504150 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:51.505772 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:51.508450 (+ 2678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:51.508458 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:51.508470 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:51.508744 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:51.509210 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:51.509212 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:51.509214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:51.509338 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:51.510242 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:51.510249 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:51.510434 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 14:52:51.510620 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 14:52:51.511212 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":146,"spinlock_wait_cycles":3968,"thread_start_us":182,"threads_started":1}
I20260620 14:52:51.515026 31797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:52:51.515942 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.515645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:34763 (local address 127.0.0.1:44878)
0620 14:52:51.515792 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.515872 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:34763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 14:52:51.516150 31745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.221.253:34763: connect: Connection refused (error 111)
I20260620 14:52:51.517211 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.516987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:34763 (local address 127.0.0.1:44892)
0620 14:52:51.517104 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.517156 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:34763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 14:52:51.517385 31797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.253:34763: Network error: Client connection negotiation failed: client connection to 127.8.221.253:34763: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.253.keytab.
Entry for principal kudu/127.8.221.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.253.keytab.
I20260620 14:52:51.521025 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.253:34763
--webserver_interface=127.8.221.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.253.keytab
--principal=kudu/127.8.221.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:38287,127.8.221.253:34763,127.8.221.252:37833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:52:51.566653 31797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } attempt: 1
I20260620 14:52:51.567893 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.567450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:34763 (local address 127.0.0.1:44908)
0620 14:52:51.567681 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.567779 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:34763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 14:52:51.568766 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.568544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:34763 (local address 127.0.0.1:44910)
0620 14:52:51.568655 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.568709 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:34763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 14:52:51.568953 31797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.253:34763: Network error: Client connection negotiation failed: client connection to 127.8.221.253:34763: connect: Connection refused (error 111)
I20260620 14:52:51.637171 31797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } attempt: 2
I20260620 14:52:51.638329 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.637934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:34763 (local address 127.0.0.1:44924)
0620 14:52:51.638139 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.638233 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:34763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 14:52:51.638125 31804 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:52:51.638520 31804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:51.638567 31804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:51.638640 31804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:52:51.639256 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.639029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:34763 (local address 127.0.0.1:44932)
0620 14:52:51.639144 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.639199 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.253:34763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 14:52:51.639446 31797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.253:34763: Network error: Client connection negotiation failed: client connection to 127.8.221.253:34763: connect: Connection refused (error 111)
W20260620 14:52:51.643504 31804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:51.643561 31804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:51.643582 31804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:51.643620 31804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:51.643656 31804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:51.649168 31804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:38287,127.8.221.253:34763,127.8.221.252:37833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.253.keytab
--principal=kudu/127.8.221.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.253:34763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.221.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:51.650341 31804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:51.651363 31804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:51.660406 31809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:51.660437 31810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:51.660490 31804 server_base.cc:1034] running on GCE node
W20260620 14:52:51.660437 31812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:51.661521 31804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:51.662652 31804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:51.663869 31804 hybrid_clock.cc:648] HybridClock initialized: now 1781967171663849 us; error 41 us; skew 500 ppm
Jun 20 14:52:51 dist-test-slave-65tz krb5kdc[31716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967171, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:51.666893 31804 init.cc:377] Logged in from keytab as kudu/127.8.221.253@KRBTEST.COM (short username kudu)
I20260620 14:52:51.667822 31804 webserver.cc:460] Webserver started at http://127.8.221.253:33097/ using document root <none> and password file <none>
I20260620 14:52:51.668418 31804 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:51.668504 31804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:51.668733 31804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:51.670853 31804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-1/data/instance:
uuid: "7d4c325efa354149a210548b011f8eb7"
format_stamp: "Formatted at 2026-06-20 14:52:51 on dist-test-slave-65tz"
server_key: "a2b835e75032a65407ffaeabaa22390f"
server_key_iv: "5b0718cc38d401e50145fadb588f1867"
server_key_version: "encryptionkey@0"
I20260620 14:52:51.671404 31804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-1/wal/instance:
uuid: "7d4c325efa354149a210548b011f8eb7"
format_stamp: "Formatted at 2026-06-20 14:52:51 on dist-test-slave-65tz"
server_key: "a2b835e75032a65407ffaeabaa22390f"
server_key_iv: "5b0718cc38d401e50145fadb588f1867"
server_key_version: "encryptionkey@0"
I20260620 14:52:51.675486 31804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:52:51.678216 31819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:51.679625 31804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:51.679775 31804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-1/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-1/wal
uuid: "7d4c325efa354149a210548b011f8eb7"
format_stamp: "Formatted at 2026-06-20 14:52:51 on dist-test-slave-65tz"
server_key: "a2b835e75032a65407ffaeabaa22390f"
server_key_iv: "5b0718cc38d401e50145fadb588f1867"
server_key_version: "encryptionkey@0"
I20260620 14:52:51.679888 31804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:51.704941 31804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:51.708246 31804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:51.708454 31804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:51.716174 31804 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.253:34763
I20260620 14:52:51.716460 31871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.253:34763 every 8 connection(s)
I20260620 14:52:51.717264 31804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-1/data/info.pb
I20260620 14:52:51.719002 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 31804
I20260620 14:52:51.719110 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-1/wal/instance
I20260620 14:52:51.719419 9079 external_mini_cluster.cc:1428] Setting key 88921fcd7a188c7e2dd5848180081325
I20260620 14:52:51.721591 31872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:51.726534 31872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.221.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.221.252@KRBTEST.COM" created.
I20260620 14:52:51.730640 31797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } attempt: 3
Jun 20 14:52:51 dist-test-slave-65tz krb5kdc[31716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967171, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.253@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
Jun 20 14:52:51 dist-test-slave-65tz krb5kdc[31716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967171, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for kudu/127.8.221.253@KRBTEST.COM
I20260620 14:52:51.744686 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.731138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:34763 (local address 127.0.0.1:44934)
0620 14:52:51.731664 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.731681 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:51.731809 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:51.732587 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:51.732591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:51.732816 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:51.732957 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:51.732966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.734443 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.734474 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:51.735362 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:51.735384 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.736623 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.736629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:51.737119 (+ 490us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.737725 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:51.737746 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:51.739740 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:51.742424 (+ 2684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:51.742427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:51.742430 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:51.742709 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:51.743092 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:51.743094 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:51.743096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:51.743190 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:51.743914 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:51.743920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:51.744029 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 14:52:51.744157 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 14:52:51.744454 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 14:52:51.745188 31800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.729074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.8.221.254:38287)
0620 14:52:51.729258 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:51.729263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:51.730331 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 14:52:51.731794 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:51.731799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:51.731844 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:51.731938 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:51.733487 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.734384 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.735766 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.736036 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.736150 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.739391 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:51.739395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:51.739397 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:51.739419 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:51.741293 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.742016 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.742021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.742023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.742139 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.742646 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.742650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.742652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.742985 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:51.743096 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:51.744578 (+ 1482us) server_negotiation.cc:299] Negotiation successful
0620 14:52:51.744717 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:52:51.745307 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.729505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38287 (local address 127.0.0.1:52276)
0620 14:52:51.730127 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.730184 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:51.731479 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:51.732159 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:51.732168 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:51.732705 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:51.733322 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:51.733334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.734528 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.734531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:51.735645 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:51.735655 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.736199 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.736203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:51.736394 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.737131 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:51.737155 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:51.739226 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:51.741428 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:51.741437 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:51.741450 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:51.741877 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:51.742272 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:51.742275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:51.742276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:51.742485 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:51.743104 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:51.743112 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:51.743509 (+ 397us) client_negotiation.cc:769] Sending connection context
0620 14:52:51.743712 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 14:52:51.744104 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":208,"threads_started":1}
I20260620 14:52:51.745609 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.731602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44934 (local address 127.8.221.253:34763)
0620 14:52:51.732058 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:51.732070 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:51.732091 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:52:51.732143 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:51.732148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:51.732392 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:51.732595 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:51.733214 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.734309 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.735557 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.736457 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.736613 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.739897 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:51.739904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:51.739918 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:51.739938 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:51.742286 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.742848 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.742856 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.742857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.742967 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.743321 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.743328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.743332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.743765 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:51.745012 (+ 1247us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:51.745155 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:52:51.745389 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"spinlock_wait_cycles":3328,"thread_start_us":185,"threads_started":1}
I20260620 14:52:51.747088 31872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:52:51.748206 31797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.221.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.252.keytab.
Entry for principal kudu/127.8.221.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.252.keytab.
I20260620 14:52:51.749081 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.748742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37833 (local address 127.0.0.1:56160)
0620 14:52:51.748907 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.748989 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 14:52:51.750001 9079 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQhiojW/build/debug/bin/kudu
/tmp/dist-test-taskQhiojW/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.252:37833
--webserver_interface=127.8.221.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.252.keytab
--principal=kudu/127.8.221.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.221.212:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.221.254:38287,127.8.221.253:34763,127.8.221.252:37833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:52:51.750195 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.749877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37833 (local address 127.0.0.1:56176)
0620 14:52:51.750050 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.750108 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260620 14:52:51.750325 31797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.252:37833: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37833: connect: Connection refused (error 111)
Jun 20 14:52:51 dist-test-slave-65tz krb5kdc[31716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967171, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.253@KRBTEST.COM for kudu/127.8.221.253@KRBTEST.COM
I20260620 14:52:51.757021 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.747964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44944 (local address 127.8.221.253:34763)
0620 14:52:51.748118 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:51.748124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:51.748249 (+ 125us) server_negotiation.cc:407] Connection header received
0620 14:52:51.748360 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:51.748363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:51.748405 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:51.748485 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:51.749003 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.749760 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.750816 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.751073 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.751180 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.753449 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:51.753453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:51.753456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:51.753477 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:51.755286 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.755764 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.755770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.755772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.755853 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.756141 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.756144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.756146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.756336 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:51.756440 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:51.756677 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 14:52:51.756842 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":55}
I20260620 14:52:51.757022 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.747758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:34763 (local address 127.0.0.1:44944)
0620 14:52:51.747886 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.748174 (+ 288us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:51.748275 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:51.748526 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:51.748530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:51.748735 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:51.748877 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:51.748886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.749900 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.749906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:51.750702 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:51.750711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.751174 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.751177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:51.751269 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.751839 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:51.751882 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:51.753321 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:51.755449 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:51.755455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:51.755457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:51.755652 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:51.755949 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:51.755952 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:51.755975 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:51.756053 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:51.756437 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:51.756440 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:51.756527 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 14:52:51.756623 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:52:51.756782 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 14:52:51.757939 31872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:52:51.758908 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.758591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37833 (local address 127.0.0.1:56178)
0620 14:52:51.758733 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.758819 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 14:52:51.759512 31821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37833: connect: Connection refused (error 111)
I20260620 14:52:51.760649 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.760439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37833 (local address 127.0.0.1:56184)
0620 14:52:51.760543 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.760595 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 14:52:51.760818 31872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.252:37833: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37833: connect: Connection refused (error 111)
I20260620 14:52:51.781551 31797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } attempt: 1
I20260620 14:52:51.782802 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.782352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37833 (local address 127.0.0.1:56192)
0620 14:52:51.782565 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.782654 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 14:52:51.783803 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.783547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37833 (local address 127.0.0.1:56202)
0620 14:52:51.783660 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.783711 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 14:52:51.783996 31797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.252:37833: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37833: connect: Connection refused (error 111)
I20260620 14:52:51.810108 31872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } attempt: 1
I20260620 14:52:51.811358 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.810856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37833 (local address 127.0.0.1:56216)
0620 14:52:51.811117 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.811225 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260620 14:52:51.812237 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.811997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37833 (local address 127.0.0.1:56224)
0620 14:52:51.812111 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.812172 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 14:52:51.812422 31872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.252:37833: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37833: connect: Connection refused (error 111)
I20260620 14:52:51.859201 31797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } attempt: 2
I20260620 14:52:51.860404 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.859988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37833 (local address 127.0.0.1:56232)
0620 14:52:51.860202 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.860305 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 14:52:51.861285 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.861065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37833 (local address 127.0.0.1:56238)
0620 14:52:51.861178 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.861226 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 14:52:51.861466 31797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.252:37833: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37833: connect: Connection refused (error 111)
W20260620 14:52:51.875581 31878 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:52:51.875937 31878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:51.875965 31878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:51.876011 31878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:51.880568 31878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:51.880653 31878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:51.880676 31878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:51.880697 31878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:51.880717 31878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:51.880676 31872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } attempt: 2
I20260620 14:52:51.881747 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.881403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37833 (local address 127.0.0.1:56252)
0620 14:52:51.881574 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.881652 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 14:52:51.882803 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.882449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37833 (local address 127.0.0.1:56254)
0620 14:52:51.882603 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.882690 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 14:52:51.883064 31872 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.221.252:37833: Network error: Client connection negotiation failed: client connection to 127.8.221.252:37833: connect: Connection refused (error 111)
I20260620 14:52:51.887449 31878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.221.212:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.221.254:38287,127.8.221.253:34763,127.8.221.252:37833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/krb5kdc/kudu_127.8.221.252.keytab
--principal=kudu/127.8.221.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.221.252:37833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.221.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 14:09:29 UTC on ccycloud.quasar-vozkmi.root.comops.site
build id 4457508
I20260620 14:52:51.888582 31878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:51.889705 31878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:51.899065 31887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:51.899118 31884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:51.899358 31878 server_base.cc:1034] running on GCE node
W20260620 14:52:51.899118 31885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:51.900053 31878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:51.901219 31878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:51.902411 31878 hybrid_clock.cc:648] HybridClock initialized: now 1781967171902379 us; error 55 us; skew 500 ppm
Jun 20 14:52:51 dist-test-slave-65tz krb5kdc[31716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967171, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:51.909375 31878 init.cc:377] Logged in from keytab as kudu/127.8.221.252@KRBTEST.COM (short username kudu)
I20260620 14:52:51.910406 31878 webserver.cc:460] Webserver started at http://127.8.221.252:35367/ using document root <none> and password file <none>
I20260620 14:52:51.910989 31878 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:51.911075 31878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:51.911309 31878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:51.913347 31878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-2/data/instance:
uuid: "1cff4899eca04f07b9b11ad09c2eb19f"
format_stamp: "Formatted at 2026-06-20 14:52:51 on dist-test-slave-65tz"
server_key: "599f78501a7e5ee17609270193b8f567"
server_key_iv: "b02969842f0e9f9a88cae018aecf31d3"
server_key_version: "encryptionkey@0"
I20260620 14:52:51.913976 31878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-2/wal/instance:
uuid: "1cff4899eca04f07b9b11ad09c2eb19f"
format_stamp: "Formatted at 2026-06-20 14:52:51 on dist-test-slave-65tz"
server_key: "599f78501a7e5ee17609270193b8f567"
server_key_iv: "b02969842f0e9f9a88cae018aecf31d3"
server_key_version: "encryptionkey@0"
I20260620 14:52:51.918212 31878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:52:51.921005 31894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:51.922547 31878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:51.922709 31878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-2/data,/tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-2/wal
uuid: "1cff4899eca04f07b9b11ad09c2eb19f"
format_stamp: "Formatted at 2026-06-20 14:52:51 on dist-test-slave-65tz"
server_key: "599f78501a7e5ee17609270193b8f567"
server_key_iv: "b02969842f0e9f9a88cae018aecf31d3"
server_key_version: "encryptionkey@0"
I20260620 14:52:51.922833 31878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:51.938854 31878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:51.942116 31878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:51.942317 31878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:51.949797 31878 rpc_server.cc:307] RPC server started. Bound to: 127.8.221.252:37833
I20260620 14:52:51.950237 31946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.221.252:37833 every 8 connection(s)
I20260620 14:52:51.950881 31878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-2/data/info.pb
I20260620 14:52:51.954865 31947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:51.958461 9079 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQhiojW/build/debug/bin/kudu as pid 31878
I20260620 14:52:51.958618 9079 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQhiojW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966871461349-9079-0/minicluster-data/master-2/wal/instance
I20260620 14:52:51.958947 9079 external_mini_cluster.cc:1428] Setting key 73b5527a305474cb5c230d2bb992df4d
I20260620 14:52:51.959820 31947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:52:51.960646 31797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } attempt: 3
Jun 20 14:52:51 dist-test-slave-65tz krb5kdc[31716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967171, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.254@KRBTEST.COM for kudu/127.8.221.252@KRBTEST.COM
Jun 20 14:52:51 dist-test-slave-65tz krb5kdc[31716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967171, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.252@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
Jun 20 14:52:51 dist-test-slave-65tz krb5kdc[31716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.252@KRBTEST.COM
I20260620 14:52:51.974241 31872 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } attempt: 3
I20260620 14:52:51.975181 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.961155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37833 (local address 127.0.0.1:56268)
0620 14:52:51.961355 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.961376 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:51.961513 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:51.963407 (+ 1894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:51.963410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:51.963661 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:51.963878 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:51.963890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.965541 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.965544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:51.966191 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:51.966200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.967118 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.967122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:51.967296 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.967909 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:51.967920 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:51.969590 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:51.973061 (+ 3471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:51.973067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:51.973071 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:51.973328 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:51.973687 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:51.973693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:51.973698 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:51.973776 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:51.974604 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:51.974608 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:51.974706 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 14:52:51.974795 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:52:51.974963 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":39}
I20260620 14:52:51.975435 31800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.962266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52282 (local address 127.8.221.254:38287)
0620 14:52:51.962469 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:51.962475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:51.962806 (+ 331us) server_negotiation.cc:407] Connection header received
0620 14:52:51.963282 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:51.963286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:51.963349 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:51.963425 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:51.964494 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.965314 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.966840 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.967107 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.967219 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.970530 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:51.970534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:51.970536 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:51.970563 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:51.972182 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.973080 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.973084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.973086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.973210 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.973632 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.973652 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.973656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.973998 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:51.974098 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:51.975049 (+ 951us) server_negotiation.cc:299] Negotiation successful
0620 14:52:51.975210 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":277,"server-negotiator.queue_time_us":107}
I20260620 14:52:51.977276 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.961429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56268 (local address 127.8.221.252:37833)
0620 14:52:51.961823 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:51.961835 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:51.961853 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:52:51.962858 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:51.962867 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:51.963224 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:51.963420 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:51.964177 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.965412 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.966316 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.966987 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.967177 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.969790 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:51.969840 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:51.969858 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:51.969886 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:51.972903 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.973485 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.973490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.973494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.973587 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.973903 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.973906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.973908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.974476 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:51.975766 (+ 1290us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:51.975829 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 14:52:51.976042 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":673,"server-negotiator.queue_time_us":289,"spinlock_wait_cycles":19072,"thread_start_us":175,"threads_started":1}
I20260620 14:52:51.977372 31950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.960711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56262 (local address 127.8.221.252:37833)
0620 14:52:51.961358 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:51.961389 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:51.961446 (+ 57us) server_negotiation.cc:407] Connection header received
0620 14:52:51.962838 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:51.962864 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:51.963228 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:51.963422 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:51.964550 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.965446 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.966783 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.967117 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.967697 (+ 580us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.970886 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:51.970893 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:51.970896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:51.970920 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:51.973653 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.974155 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.974158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.974160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.974237 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.974547 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.974552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.974555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.974735 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:51.975534 (+ 799us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:51.975680 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 14:52:51.976270 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":329,"server-negotiator.queue_time_us":438,"spinlock_wait_cycles":17280,"thread_start_us":180,"threads_started":1}
I20260620 14:52:51.977273 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.962226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.254:38287 (local address 127.0.0.1:52282)
0620 14:52:51.962655 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.962699 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:51.963046 (+ 347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:51.963622 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:51.963628 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:51.964082 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:51.964349 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:51.964361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.965441 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.965445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:51.966731 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:51.966743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.967369 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.967375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:51.967511 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.968280 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:51.968301 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:51.970369 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:51.972303 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:51.972315 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:51.972326 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:51.972954 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:51.973312 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:51.973316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:51.973318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:51.973470 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:51.974121 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:51.974128 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:51.974528 (+ 400us) client_negotiation.cc:769] Sending connection context
0620 14:52:51.975645 (+ 1117us) client_negotiation.cc:240] Negotiation successful
0620 14:52:51.976172 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":128,"thread_start_us":99,"threads_started":1}
I20260620 14:52:51.979516 31947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:52:51 dist-test-slave-65tz krb5kdc[31716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967171, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.253@KRBTEST.COM for kudu/127.8.221.252@KRBTEST.COM
I20260620 14:52:51.983403 31797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d: Bootstrap starting.
Jun 20 14:52:51 dist-test-slave-65tz krb5kdc[31716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967171, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.252@KRBTEST.COM for kudu/127.8.221.253@KRBTEST.COM
I20260620 14:52:51.986856 31797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:51.986982 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.974728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37833 (local address 127.0.0.1:56270)
0620 14:52:51.974840 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.974857 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:51.974932 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:51.976391 (+ 1459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:51.976394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:51.976590 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:51.976708 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:51.976715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.977841 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.977845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:51.978395 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:51.978402 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.979338 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.979342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:51.979451 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.980489 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:51.980502 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:51.982264 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:51.984543 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:51.984547 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:51.984549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:51.984836 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:51.985738 (+ 902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:51.985742 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:51.985745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:51.985906 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:51.986413 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:51.986419 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:51.986518 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 14:52:51.986601 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:52:51.986768 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:52:51.987730 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.975000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56270 (local address 127.8.221.252:37833)
0620 14:52:51.976143 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:51.976158 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:51.976172 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:52:51.976208 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:51.976210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:51.976251 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:51.976817 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:51.976902 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.977691 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.978513 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.979184 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.979291 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.982423 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:51.982427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:51.982430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:51.982454 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:51.984414 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.985527 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.985531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.985532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.985618 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.986034 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.986038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.986040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.986297 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:51.987304 (+ 1007us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:51.987386 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:52:51.987514 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1057,"thread_start_us":120,"threads_started":1}
I20260620 14:52:51.987865 31797 log.cc:824] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:51.990842 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.980052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.253:34763 (local address 127.0.0.1:44958)
0620 14:52:51.981158 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.981173 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:51.981265 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:51.981544 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:51.981547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:51.981710 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:51.981862 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:51.981873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.982900 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.982904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:51.983721 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:51.983731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.984361 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.984365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:51.984464 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.984928 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:51.984943 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:51.986509 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:51.989194 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:51.989197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:51.989199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:51.989419 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:51.989718 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:51.989721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:51.989723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:51.989831 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:51.990376 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:51.990379 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:51.990470 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 14:52:51.990560 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:52:51.990706 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028}
I20260620 14:52:51.991466 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.980144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44958 (local address 127.8.221.253:34763)
0620 14:52:51.980259 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:51.980265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:51.981264 (+ 999us) server_negotiation.cc:407] Connection header received
0620 14:52:51.981368 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:51.981371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:51.981418 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:51.981518 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:51.982055 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.982790 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.983850 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.984229 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.984375 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.986665 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:51.986668 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:51.986671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:51.986692 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:51.989067 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.989528 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.989531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.989533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.989621 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.989941 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.989948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.989952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.990189 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:51.990324 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:51.991211 (+ 887us) server_negotiation.cc:299] Negotiation successful
0620 14:52:51.991335 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 14:52:51 dist-test-slave-65tz krb5kdc[31716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.254@KRBTEST.COM
I20260620 14:52:51.991444 31872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7: Bootstrap starting.
I20260620 14:52:51.992079 31797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d: No bootstrap required, opened a new log
I20260620 14:52:51.994421 31872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:51.994640 31947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:52:51.995677 31872 log.cc:824] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:51.996009 31800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.980491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52286 (local address 127.8.221.254:38287)
0620 14:52:51.980609 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:51.980613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:51.986169 (+ 5556us) server_negotiation.cc:407] Connection header received
0620 14:52:51.986209 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:51.986212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:51.986271 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:51.986338 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:51.987255 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.987970 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.988877 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.989099 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.989201 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.992090 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:51.992096 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:51.992101 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:51.992127 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:51.993834 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.994579 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.994583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.994585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.994667 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:51.994974 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:51.994979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:51.994981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:51.995192 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:51.995363 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:51.995726 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 14:52:51.995847 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:52:51.997133 31797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } } peers { permanent_uuid: "7d4c325efa354149a210548b011f8eb7" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } } peers { permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } }
I20260620 14:52:51.997407 31797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fcd21ceab624a4ebd3884a81ae4f50d, State: Initialized, Role: FOLLOWER
I20260620 14:52:51.997859 31797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } } peers { permanent_uuid: "7d4c325efa354149a210548b011f8eb7" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } } peers { permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } }
I20260620 14:52:51.998420 31872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7: No bootstrap required, opened a new log
I20260620 14:52:51.999588 31797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 14:52:51 dist-test-slave-65tz krb5kdc[31716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967171, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.221.252@KRBTEST.COM for kudu/127.8.221.252@KRBTEST.COM
I20260620 14:52:52.003963 31872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } } peers { permanent_uuid: "7d4c325efa354149a210548b011f8eb7" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } } peers { permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } }
I20260620 14:52:52.004284 31872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4c325efa354149a210548b011f8eb7, State: Initialized, Role: FOLLOWER
Jun 20 14:52:51 dist-test-slave-65tz krb5kdc[31716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.221.253@KRBTEST.COM
I20260620 14:52:52.004828 31872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } } peers { permanent_uuid: "7d4c325efa354149a210548b011f8eb7" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } } peers { permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } }
I20260620 14:52:52.006400 31872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:52:52.006597 31967 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:52:52.006701 31967 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:52:52.009555 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.997862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44960 (local address 127.8.221.253:34763)
0620 14:52:51.997986 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:51.997992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:51.998203 (+ 211us) server_negotiation.cc:407] Connection header received
0620 14:52:51.998392 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:51.998395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:51.998443 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:51.998492 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:51.999330 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:52.000050 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:52.001206 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:52.001601 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:52.001685 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:52.005076 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:52.005080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:52.005083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:52.005110 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:52.007276 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:52.007833 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:52.007837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:52.007839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:52.007931 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:52.008282 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:52.008287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:52.008288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:52.008566 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:52.008781 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:52.009204 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 14:52:52.009369 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:52:52.005343 31729 master_runner.cc:186] Error getting master registration for 127.8.221.253:34763: OK, instance_id { permanent_uuid: "7d4c325efa354149a210548b011f8eb7" instance_seqno: 1781967171713998 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:52:52.010756 31956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } } peers { permanent_uuid: "7d4c325efa354149a210548b011f8eb7" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } } peers { permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } } }
I20260620 14:52:52.010919 31956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:52:52.013118 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.995099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.221.252:37833 (local address 127.0.0.1:56274)
0620 14:52:51.995192 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:51.995207 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:51.995282 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:51.996394 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:51.996397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:51.996580 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:51.996709 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:51.996716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.997707 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.997710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:51.998445 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:51.998455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.999028 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.999031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:51.999102 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.999622 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:51.999637 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:52.001192 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:52.011543 (+ 10351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:52.011546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:52.011548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:52.011798 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:52.012091 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:52.012096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:52.012098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:52.012194 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:52.012634 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:52.012637 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:52.012716 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 14:52:52.012800 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:52:52.012952 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 14:52:52.013727 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.995247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.8.221.252:37833)
0620 14:52:51.996188 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:51.996193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:51.996206 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:52:51.996237 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:51.996240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:51.996282 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:51.996815 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:51.996881 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.997596 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.998577 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.998912 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.999334 (+ 422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:52.001332 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:52.001335 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:52.001338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:52.001359 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:52.011422 (+ 10063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:52.011913 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:52.011916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:52.011920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:52.012002 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:52.012292 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:52.012294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:52.012296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:52.012536 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:52.013414 (+ 878us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:52.013481 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 14:52:52.013598 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857}
I20260620 14:52:52.009754 31729 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:52:52.016723 31947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f: Bootstrap starting.
W20260620 14:52:52.017751 31980 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:52:52.017992 31980 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:52:52.019387 31947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:52.015151 31804 master_runner.cc:186] Error getting master registration for 127.8.221.252:37833: OK, instance_id { permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" instance_seqno: 1781967171947716 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:52:52.019538 31804 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:52:52.020221 31947 log.cc:824] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:52.020257 9079 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:52.020958 31969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } } peers { permanent_uuid: "7d4c325efa354149a210548b011f8eb7" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } } peers { permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } } }
I20260620 14:52:52.021175 31969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:52:52.021970 31771 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52286
I20260620 14:52:52.022298 31771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:52:52.022433 31771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:52.023725 31947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f: No bootstrap required, opened a new log
I20260620 14:52:52.024044 31771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } } peers { permanent_uuid: "7d4c325efa354149a210548b011f8eb7" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } } peers { permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } }
I20260620 14:52:52.025193 31771 leader_election.cc:288] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [CANDIDATE]: Term 1 election: Requested vote from peers 7d4c325efa354149a210548b011f8eb7 (127.8.221.253:34763), 1cff4899eca04f07b9b11ad09c2eb19f (127.8.221.252:37833)
I20260620 14:52:52.026324 31846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7d4c325efa354149a210548b011f8eb7"
I20260620 14:52:52.026592 31846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:52.028084 31846 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fcd21ceab624a4ebd3884a81ae4f50d in term 1.
I20260620 14:52:52.028664 31745 leader_election.cc:302] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fcd21ceab624a4ebd3884a81ae4f50d, 7d4c325efa354149a210548b011f8eb7; no voters:
I20260620 14:52:52.028622 31947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } } peers { permanent_uuid: "7d4c325efa354149a210548b011f8eb7" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } } peers { permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } }
I20260620 14:52:52.028894 31947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cff4899eca04f07b9b11ad09c2eb19f, State: Initialized, Role: FOLLOWER
I20260620 14:52:52.029251 31947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } } peers { permanent_uuid: "7d4c325efa354149a210548b011f8eb7" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } } peers { permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } }
I20260620 14:52:52.029913 31956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:52.030380 31987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } } peers { permanent_uuid: "7d4c325efa354149a210548b011f8eb7" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } } peers { permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } } }
I20260620 14:52:52.030646 31987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:52:52.030876 31947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:52.030915 31921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1cff4899eca04f07b9b11ad09c2eb19f"
I20260620 14:52:52.031129 31921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:52.032636 31921 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fcd21ceab624a4ebd3884a81ae4f50d in term 1.
I20260620 14:52:52.030825 31956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [term 1 LEADER]: Becoming Leader. State: Replica: 3fcd21ceab624a4ebd3884a81ae4f50d, State: Running, Role: LEADER
I20260620 14:52:52.034198 31956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } } peers { permanent_uuid: "7d4c325efa354149a210548b011f8eb7" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } } peers { permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } }
W20260620 14:52:52.037096 31999 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:52:52.037204 31999 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:52:52.043546 31988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fcd21ceab624a4ebd3884a81ae4f50d. Latest consensus state: current_term: 1 leader_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } } peers { permanent_uuid: "7d4c325efa354149a210548b011f8eb7" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } } peers { permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } } }
I20260620 14:52:52.043834 31988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:52.058471 32003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:52.059672 32003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:52.059834 31800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:52.044860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52288 (local address 127.8.221.254:38287)
0620 14:52:52.044962 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:52.044966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:52.046064 (+ 1098us) server_negotiation.cc:407] Connection header received
0620 14:52:52.046230 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:52.046233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:52.046285 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:52.046339 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:52.047395 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:52.048118 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:52.049325 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:52.049659 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:52.049734 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:52.054701 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:52.054705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:52.054708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:52.054736 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:52.056661 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:52.057187 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:52.057191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:52.057195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:52.057286 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:52.057572 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:52.057576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:52.057578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:52.057850 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:52.057967 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:52.059542 (+ 1575us) server_negotiation.cc:299] Negotiation successful
0620 14:52:52.059684 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:52:52.067553 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:52.034893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56278 (local address 127.8.221.252:37833)
0620 14:52:52.035012 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:52.035017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:52.044388 (+ 9371us) server_negotiation.cc:407] Connection header received
0620 14:52:52.044546 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:52.044551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:52.044607 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:52.044687 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:52.049254 (+ 4567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:52.050004 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:52.057254 (+ 7250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:52.057637 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:52.057718 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:52.060380 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:52.060383 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:52.060386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:52.060413 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:52.062201 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:52.065052 (+ 2851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:52.065056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:52.065058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:52.065148 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:52.066540 (+ 1392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:52.066544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:52.066546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:52.066797 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:52.066915 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:52.067255 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 14:52:52.067385 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:52:52.072955 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:52.045504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44966 (local address 127.8.221.253:34763)
0620 14:52:52.045614 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:52.045618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:52.051777 (+ 6159us) server_negotiation.cc:407] Connection header received
0620 14:52:52.051935 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:52.051938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:52.051989 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:52.052038 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:52.052788 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:52.053535 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:52.063927 (+ 10392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:52.064291 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:52.064373 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:52.069348 (+ 4975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:52.069352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:52.069355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:52.069387 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:52.071152 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:52.071635 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:52.071651 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:52.071653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:52.071736 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:52.072014 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:52.072018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:52.072019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:52.072232 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:52.072342 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:52.072671 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 14:52:52.072790 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:52:52.079586 31921 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f [term 1 FOLLOWER]: Refusing update from remote peer 3fcd21ceab624a4ebd3884a81ae4f50d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:52.080350 31956 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:52.084893 31987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fcd21ceab624a4ebd3884a81ae4f50d. Latest consensus state: current_term: 1 leader_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } } peers { permanent_uuid: "7d4c325efa354149a210548b011f8eb7" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } } peers { permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } } }
I20260620 14:52:52.085069 31987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:52:52.089057 31846 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7 [term 1 FOLLOWER]: Refusing update from remote peer 3fcd21ceab624a4ebd3884a81ae4f50d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:52.089754 31956 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4c325efa354149a210548b011f8eb7" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:52.098721 31969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fcd21ceab624a4ebd3884a81ae4f50d. Latest consensus state: current_term: 1 leader_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } } peers { permanent_uuid: "7d4c325efa354149a210548b011f8eb7" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } } peers { permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } } }
I20260620 14:52:52.099020 31969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:52:52.111748 31956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } } peers { permanent_uuid: "7d4c325efa354149a210548b011f8eb7" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } } peers { permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } } }
I20260620 14:52:52.111953 31956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:52.113468 31969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } } peers { permanent_uuid: "7d4c325efa354149a210548b011f8eb7" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } } peers { permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } } }
I20260620 14:52:52.113623 31969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:52:52.114140 32006 mvcc.cc:205] Tried to move back new op lower bound from 7298937536824619008 to 7298937536679174144. Current Snapshot: MvccSnapshot[applied={T|T < 7298937536824619008}]
I20260620 14:52:52.118050 31988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } } peers { permanent_uuid: "7d4c325efa354149a210548b011f8eb7" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } } peers { permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } } }
I20260620 14:52:52.118202 31988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:52.119531 31987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fcd21ceab624a4ebd3884a81ae4f50d" member_type: VOTER last_known_addr { host: "127.8.221.254" port: 38287 } } peers { permanent_uuid: "7d4c325efa354149a210548b011f8eb7" member_type: VOTER last_known_addr { host: "127.8.221.253" port: 34763 } } peers { permanent_uuid: "1cff4899eca04f07b9b11ad09c2eb19f" member_type: VOTER last_known_addr { host: "127.8.221.252" port: 37833 } } }
I20260620 14:52:52.119675 31987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:52:52.120795 32009 mvcc.cc:205] Tried to move back new op lower bound from 7298937536824619008 to 7298937536679174144. Current Snapshot: MvccSnapshot[applied={T|T < 7298937536824619008}]
I20260620 14:52:52.120986 32007 mvcc.cc:205] Tried to move back new op lower bound from 7298937536824619008 to 7298937536679174144. Current Snapshot: MvccSnapshot[applied={T|T < 7298937536824619008}]
I20260620 14:52:52.126539 32003 catalog_manager.cc:1347] Generated new cluster ID: d249a4dca4a045099302d8d80d0fe82b
I20260620 14:52:52.126655 32003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:52.145321 32003 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:52.146886 32003 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:52.185959 32003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fcd21ceab624a4ebd3884a81ae4f50d: Generated new TSK 0
I20260620 14:52:52.187155 32003 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:52:52.227603 31953 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:52.225118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.8.221.252:37833)
0620 14:52:52.225275 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:52.225281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:52.226064 (+ 783us) server_negotiation.cc:407] Connection header received
0620 14:52:52.226355 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:52.226359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:52.226424 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:52.226508 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:52.227421 (+ 913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56292: BlockingRecv error: recv got EOF from 127.0.0.1:56292 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 14:52:52.227766 31800 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:52.225467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52298 (local address 127.8.221.254:38287)
0620 14:52:52.225592 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:52.225597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:52.226731 (+ 1134us) server_negotiation.cc:407] Connection header received
0620 14:52:52.226927 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:52.226931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:52.226978 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:52.227028 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:52.227619 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52298: BlockingRecv error: recv got EOF from 127.0.0.1:52298 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 14:52:52.229319 31876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:52.226399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.8.221.253:34763)
0620 14:52:52.226552 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:52.226556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:52.228453 (+ 1897us) server_negotiation.cc:407] Connection header received
0620 14:52:52.228618 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:52.228621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:52.228683 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:52.228732 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:52.229199 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44982: BlockingRecv error: recv got EOF from 127.0.0.1:44982 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 14:52:52.233386 31876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:52.232399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.8.221.253:34763)
0620 14:52:52.232526 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:52.232530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:52.232543 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:52:52.232618 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:52.232620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:52.232694 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:52.232777 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:52.233292 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44984: BlockingRecv error: recv got EOF from 127.0.0.1:44984 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 14:52:52.238060 31876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:52.237117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.8.221.253:34763)
0620 14:52:52.237253 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:52.237257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:52.237271 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:52:52.237312 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:52.237315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:52.237358 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:52.237431 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:52.237968 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44986: BlockingRecv error: recv got EOF from 127.0.0.1:44986 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 14:52:52.243801 31876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:52.242631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44998 (local address 127.8.221.253:34763)
0620 14:52:52.242751 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:52.242755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:52.242921 (+ 166us) server_negotiation.cc:407] Connection header received
0620 14:52:52.243066 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:52.243070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:52.243129 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:52.243244 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:52.243678 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44998: BlockingRecv error: recv got EOF from 127.0.0.1:44998 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 14:52:52.251232 31876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:52.249915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45004 (local address 127.8.221.253:34763)
0620 14:52:52.250066 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:52.250071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:52.250191 (+ 120us) server_negotiation.cc:407] Connection header received
0620 14:52:52.250314 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:52.250317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:52.250372 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:52.250469 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:52.251112 (+ 643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45004: BlockingRecv error: recv got EOF from 127.0.0.1:45004 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 14:52:52.262850 31876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:52.261497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.8.221.253:34763)
0620 14:52:52.261678 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:52.261685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:52.261984 (+ 299us) server_negotiation.cc:407] Connection header received
0620 14:52:52.262107 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:52.262110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:52.262164 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:52.262251 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:52.262733 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45012: BlockingRecv error: recv got EOF from 127.0.0.1:45012 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 14:52:52.282820 31876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:52.281438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45024 (local address 127.8.221.253:34763)
0620 14:52:52.281652 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:52.281657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:52.281749 (+ 92us) server_negotiation.cc:407] Connection header received
0620 14:52:52.281963 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:52.281967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:52.282027 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:52.282109 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:52.282672 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45024: BlockingRecv error: recv got EOF from 127.0.0.1:45024 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260620 14:52:52.318674 31876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:52.317207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45034 (local address 127.8.221.253:34763)
0620 14:52:52.317383 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:52.317388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:52.317589 (+ 201us) server_negotiation.cc:407] Connection header received
0620 14:52:52.317897 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:52.317902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:52.317969 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:52.318085 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:52.318535 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45034: BlockingRecv error: recv got EOF from 127.0.0.1:45034 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 14:52:52.386920 31876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:52.385244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45050 (local address 127.8.221.253:34763)
0620 14:52:52.385444 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:52.385449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:52.385736 (+ 287us) server_negotiation.cc:407] Connection header received
0620 14:52:52.385942 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:52.385946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:52.386011 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:52.386098 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:52.386755 (+ 657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45050: BlockingRecv error: recv got EOF from 127.0.0.1:45050 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 14:52:52.518895 31876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:52.517408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45052 (local address 127.8.221.253:34763)
0620 14:52:52.517603 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:52.517610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:52.517786 (+ 176us) server_negotiation.cc:407] Connection header received
0620 14:52:52.517989 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:52.517993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:52.518054 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:52.518194 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:52.518747 (+ 553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45052: BlockingRecv error: recv got EOF from 127.0.0.1:45052 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 14:52:52.779842 31876 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:52:52.777851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.8.221.253:34763)
0620 14:52:52.778193 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:52.778199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:52.778293 (+ 94us) server_negotiation.cc:407] Connection header received
0620 14:52:52.778566 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:52.778571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:52.778675 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:52.778782 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:52.779712 (+ 930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45060: BlockingRecv error: recv got EOF from 127.0.0.1:45060 (error 108)
Metrics: {"server-negotiator.queue_time_us":153}
I20260620 14:52:53.020323 31980 catalog_manager.cc:1259] Loaded cluster ID: d249a4dca4a045099302d8d80d0fe82b
I20260620 14:52:53.020447 31980 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7: loading cluster ID for follower catalog manager: success
I20260620 14:52:53.023180 31980 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7: acquiring CA information for follower catalog manager: success
I20260620 14:52:53.024700 31980 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7d4c325efa354149a210548b011f8eb7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:52:53.038689 31999 catalog_manager.cc:1259] Loaded cluster ID: d249a4dca4a045099302d8d80d0fe82b
I20260620 14:52:53.038798 31999 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f: loading cluster ID for follower catalog manager: success
I20260620 14:52:53.040509 31999 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f: acquiring CA information for follower catalog manager: success
I20260620 14:52:53.041689 31999 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1cff4899eca04f07b9b11ad09c2eb19f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:52:53.298730 32078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:53.294595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.8.221.253:34763)
0620 14:52:53.294855 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:53.294860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:53.295030 (+ 170us) server_negotiation.cc:407] Connection header received
0620 14:52:53.295247 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:53.295250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:53.295374 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:53.295490 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:53.295746 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:53.296277 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:53.297168 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:53.297397 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:53.297509 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:53.297861 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:53.298318 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:53.298451 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:53.298491 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 14:52:53.298550 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":100,"threads_started":1}
I20260620 14:52:53.305891 32084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:53.302219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56300 (local address 127.8.221.252:37833)
0620 14:52:53.302543 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:53.302548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:53.302562 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:52:53.302651 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:53.302653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:53.302765 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:53.302885 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:53.303090 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:53.303597 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:53.304428 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:53.304667 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:53.304770 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:53.305053 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:53.305446 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:53.305574 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:53.305643 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 14:52:53.305694 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":116,"threads_started":1}
I20260620 14:52:53.314237 32078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:53.310034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45066 (local address 127.8.221.253:34763)
0620 14:52:53.310234 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:53.310238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:53.310459 (+ 221us) server_negotiation.cc:407] Connection header received
0620 14:52:53.310645 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:53.310648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:53.310700 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:53.310775 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:53.311152 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:53.311950 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:53.313057 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:53.313301 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:53.313423 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:53.313663 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:53.313752 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:53.313855 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:53.313969 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 14:52:53.314031 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 14:52:53.314587 32084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:53.309691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56314 (local address 127.8.221.252:37833)
0620 14:52:53.309855 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:53.309862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:53.310343 (+ 481us) server_negotiation.cc:407] Connection header received
0620 14:52:53.310483 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:53.310486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:53.310552 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:53.310638 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:53.311001 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:53.311861 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:53.313121 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:53.313453 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:53.313547 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:53.314049 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:53.314139 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:53.314203 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:53.314349 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 14:52:53.314430 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:52:53.315343 32090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:53.309546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52306 (local address 127.8.221.254:38287)
0620 14:52:53.309994 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:53.310001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:53.310260 (+ 259us) server_negotiation.cc:407] Connection header received
0620 14:52:53.310404 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:53.310409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:53.310635 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:53.310771 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:53.311110 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:53.311665 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:53.313077 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:53.313419 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:53.313529 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:53.314104 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:53.314456 (+ 352us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:53.314542 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:53.314679 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 14:52:53.315190 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":116,"threads_started":1}
I20260620 14:52:53.318634 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 31729
I20260620 14:52:53.325685 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 31804
I20260620 14:52:53.333212 9079 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQhiojW/build/debug/bin/kudu with pid 31878
2026-06-20T14:52:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3274 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6455 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (301867 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST